Method, device, electronic device and storage medium for generating panoramic image

Abstract

The present disclosure provides a method, device, electronic device and storage medium for generating a panoramic image, which relates to the field of image processing technology, and in particular to the field of computer vision. The specific implementation scheme is: using a panoramic camera component to collect multiple images; obtaining estimated parameter information corresponding to the multiple images, wherein the estimated parameter information is used to estimate the expansion effect and position information of the expansion image corresponding to each of the multiple images; using a spherical projection method to expand the multiple images to obtain the image expansion results; using a spherical splicing method to splice the image expansion results to generate a target panoramic image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, further to the field of computer vision, and in particular to a method, device, electronic device and storage medium for generating a panoramic image. Background Art

[0002] Panoramic images can show as much of the surrounding environment as possible through wide-angle expressions and paintings, photos, videos, three-dimensional models, etc.

[0003] In a related solution, image materials are obtained through shooting hardware, and multiple image materials are imported into panoramic software for automatic stitching to obtain a stitched panoramic image. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device and storage medium for generating a panoramic image, so as to improve the image stitching effect and stitching efficiency when generating a panoramic image.

[0005] According to one aspect of the present disclosure, a method for generating a panoramic image is provided, comprising: collecting multiple images using a panoramic camera component; obtaining estimated parameter information corresponding to the multiple images, wherein the estimated parameter information is used to estimate an expansion effect of an expanded image corresponding to each of the multiple images and position information of the expanded image; performing expansion processing on the multiple images using a spherical projection method to obtain image expansion results; and performing stitching processing on the image expansion results using a spherical stitching method to generate a target panoramic image.

[0006] According to another aspect of the present disclosure, there is provided a device for generating a panoramic image, comprising: a collection module for collecting multiple images using a panoramic camera assembly; an acquisition module for acquiring estimated parameter information corresponding to the multiple images, wherein the estimated parameter information is used to estimate the expansion effect and position of the expanded image corresponding to each of the multiple images; an expansion module for expanding the multiple images by adopting a spherical projection method to obtain image expansion results; and a generation module for splicing the image expansion results by adopting a spherical splicing method to generate a target panoramic image.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a panoramic image proposed in the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method for generating a panoramic image proposed in the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When a processor executes the method for generating a panoramic image proposed in the present disclosure, the computer program executes the method.

[0010] In the present disclosure, a plurality of pictures are collected by utilizing a panoramic camera component; estimated parameter information corresponding to the plurality of pictures is obtained, wherein the estimated parameter information is used to estimate the expansion effect of the expansion image corresponding to each picture in the plurality of pictures and the position information of the expansion image; a spherical projection method is used to expand the plurality of pictures to obtain picture expansion results; a spherical stitching method is used to stitch the picture expansion results to generate a target panoramic picture, thereby achieving the purpose of quickly generating a panoramic picture by utilizing the plurality of pictures collected by the panoramic photography component, and being able to improve the picture stitching effect and stitching efficiency when generating a panoramic picture.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating a panoramic image according to an embodiment of the present disclosure;

[0014] Figure 2 is a flow chart of a method for generating a panoramic image according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of a process of generating a panoramic image according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram of an input picture according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of a black border of a picture according to an embodiment of the present disclosure;

[0018] Figure 6 is a panorama generated according to an embodiment of the present disclosure;

[0019] Figure 7 It is a structural block diagram of a device for generating a panoramic image according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0022] In a related solution, when the image material obtained by the shooting hardware is a fisheye picture, it is difficult to obtain a panoramic picture with a good stitching effect by using panoramic software processing due to the large distortion of the fisheye picture.

[0023] According to an embodiment of the present disclosure, a method for generating a panoramic image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] The method embodiments provided by the embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal or a similar electronic device. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein. Figure 1 The present invention shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for generating a panoramic image.

[0025] like Figure 1As shown, the computer terminal 100 includes a computing unit 101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 102 or a computer program loaded from a storage unit 108 into a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the computer terminal 100 can also be stored. The computing unit 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0026] A number of components in the computer terminal 100 are connected to the I / O interface 105, including: an input unit 106, such as a keyboard, a mouse, etc.; an output unit 107, such as various types of displays, speakers, etc.; a storage unit 108, such as a disk, an optical disk, etc.; and a communication unit 109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 109 allows the computer terminal 100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0027] The computing unit 101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 101 performs the method for generating a panoramic image described herein. For example, in some embodiments, the method for generating a panoramic image may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 108. In some embodiments, part or all of the computer program may be loaded and / or installed on the computer terminal 100 via the ROM 102 and / or the communication unit 109. When the computer program is loaded into the RAM 103 and executed by the computing unit 101, one or more steps of the method for generating a panoramic image described herein may be performed. Alternatively, in other embodiments, the computing unit 101 may be configured to perform the method for generating a panoramic image in any other appropriate manner (e.g., by means of firmware).

[0028] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0029] It should be noted that, in some optional embodiments, the above Figure 1 The electronic device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the electronic devices described above.

[0030] Under the above operating environment, the present disclosure provides Figure 2 The method for generating a panoramic image shown in FIG. 1 can be performed by Figure 1 The computer terminal or similar electronic device shown is used for execution. Figure 2 FIG. 1 is a flow chart of a method for generating a panoramic image according to an embodiment of the present disclosure. Figure 2 As shown, the method may include the following steps:

[0031] Step S20, collecting multiple pictures using a panoramic camera assembly;

[0032] Optionally, the panoramic photography component may be packaged in an image acquisition device in a mobile terminal, or may be a detachable independent device that can be assembled on a rear camera of the mobile terminal for image acquisition.

[0033] Optionally, a panoramic camera acquisition component is used to rotate and capture multiple pictures, and the captured multiple pictures can be used to stitch together to generate a panoramic image.

[0034] For example, the panoramic photography component can be a fisheye lens, the focal length of the fisheye lens is less than or equal to 16mm, the viewing angle is close to or equal to 180°, and the fisheye lens can be used to collect multiple fisheye pictures. The fisheye lens is added to the rear camera of the mobile phone, and the mobile phone is placed on the Bluetooth rotating gimbal, and the mobile phone is controlled to shoot when the gimbal rotates, thereby collecting multiple pictures.

[0035] Step S22, obtaining estimated parameter information corresponding to the multiple pictures, wherein the estimated parameter information is used to estimate the expansion effect of the expansion image corresponding to each picture in the multiple pictures and the position information of the expansion image;

[0036] The above-mentioned estimated parameter information includes: panoramic camera component model parameters and rotation matrix.

[0037] Specifically, the implementation process of obtaining the estimated parameter information corresponding to multiple images can refer to the further introduction of the embodiment of the present disclosure, which will not be described in detail.

[0038] Step S26, using spherical projection to expand the multiple images to obtain image expansion results;

[0039] Optionally, a spherical projection method is used to expand the multiple images.

[0040] Step S28, using a spherical stitching method to stitch the image expansion results to generate a target panoramic image.

[0041] Specifically, the spherical stitching method is used to stitch the expanded image results to generate a target panoramic image. Please refer to the further introduction of the embodiment of the present disclosure, which will not be described in detail.

[0042] According to the above steps S20 to S24 of the present disclosure, multiple pictures are collected by using a panoramic camera component; estimated parameter information corresponding to the multiple pictures is obtained, wherein the estimated parameter information is used to estimate the expansion effect of the expansion image corresponding to each picture in the multiple pictures and the position information of the expansion image; the multiple pictures are expanded by using a spherical projection method to obtain the picture expansion results; the picture expansion results are stitched by using a spherical stitching method to generate a target panoramic picture, thereby achieving the purpose of quickly generating a panoramic picture by using multiple pictures collected by the panoramic photography component, and being able to improve the picture stitching effect and stitching efficiency when generating a panoramic picture.

[0043] The method for generating a panoramic image in the above embodiment is further introduced below.

[0044] As an optional implementation manner, in step S22, obtaining estimated parameter information corresponding to multiple pictures includes:

[0045] Step S220, performing graphic detection on each of the multiple pictures to obtain a target graphic corresponding to each picture;

[0046] Optionally, performing graphic detection on each of the multiple images includes performing circle detection on each of the multiple images. The target graphic corresponding to each of the multiple images may be a circle that is finally detected.

[0047] Specifically, the implementation process of performing circle detection on each of the multiple pictures and obtaining the target graphic corresponding to each picture can refer to the further introduction of the embodiment below, which is not described in detail.

[0048] Step S222, perform feature detection on the target graphic to extract multiple feature points, and perform pose estimation based on the multiple feature points to determine the estimated parameter information, or determine the estimated parameter information using angular motion detection data collected by the angular motion detection component, wherein the angular motion detection data is detection data recorded synchronously when a panoramic camera component is used to collect multiple images.

[0049] Optionally, feature detection is performed on the target graphic to obtain multiple feature descriptors, and multiple feature points are extracted from the multiple feature descriptors, wherein each feature descriptor is a vector for describing a corresponding feature point.

[0050] Specifically, the pose estimation based on multiple feature points and the implementation process of determining the estimated parameter information are as follows: the extracted multiple feature points are projected into the fisheye hemisphere model, and then the rotation matrix is ​​obtained by using the random sampling consistency. Among them, the fisheye hemisphere model is a half spherical surface of a unit sphere, and the relationship between the multiple feature points on the fisheye image in the world coordinate system is also the rotation relationship of their 3D points on the fisheye hemisphere model, that is, the rotation matrix. The feature points are projected into the fisheye hemisphere model, and the rotation matrix can be accurately estimated. Furthermore, the projection method of the fisheye model and the calculation method of the reprojection error are determined, and the optimization of the fisheye camera model parameters and the rotation matrix is ​​completed using the bundle adjustment optimization algorithm, thereby obtaining more accurate estimated parameter information.

[0051] Specifically, the implementation process of determining the estimated parameter information using the angular motion detection data collected by the angular motion component is as follows: initialize the camera internal parameters, use the circle detection in the above step S220 to obtain the radius of the circle center, set the field of view (FOV) to the viewing angle of the fisheye camera, generally set to 180°, initialize the parameters of the fisheye camera model to: a=0, b=-0.2, c=0, and use the Euler angle collected by the gyroscope to directly calculate the rotation matrix. The Euler angle includes the pitch angle, yaw angle and roll angle. Among them, the projection function of the fisheye lens is shown in the following formula (1):

[0052] r(θ)≈a*θ+b*θ3 +c*θ 5 Formula (1)

[0053] Optionally, when pose estimation based on multiple feature points fails, the angular motion detection data collected by the angular motion detection component is used to determine the estimated parameter information. Specifically, if the bundle adjustment optimization algorithm has converged and the estimated parameter value is within a reasonable range, the pose estimation is deemed to be successful, otherwise it is deemed to be a failure.

[0054] Based on the above steps S220 to S222, graphic detection can be performed on each of the multiple pictures to obtain the target graphic corresponding to each picture, and further feature detection and extraction can be performed on the target graphic. The pose estimation can be performed using the multiple feature points obtained by extraction to determine the estimated parameter information, or the estimated parameter information can be determined using the angular motion acquisition data collected by the angular motion detection component. Accurate estimated parameter information can be obtained for subsequent image processing.

[0055] As an optional implementation, the estimated parameter information includes: panoramic camera component model parameters and a rotation matrix. The panoramic camera component model parameters are used to estimate the expansion effect of the expanded image corresponding to each of the multiple images, and the rotation matrix is ​​used to estimate the position of the expanded image in the target panoramic image.

[0056] The above panoramic camera component model is a fisheye camera model parameter.

[0057] As an optional implementation, in step S220, performing graphic detection on each of the multiple pictures to obtain a target graphic corresponding to each picture includes:

[0058] Step S2200, adjusting the brightness and contrast of each of the multiple pictures by using color scale adjustment to obtain an adjustment result;

[0059] Specifically, adjusting the brightness and contrast of each of the multiple images by using the color scale adjustment includes: reducing the brightness of each of the multiple images, and increasing the contrast of each of the multiple images. The color scale adjustment includes adjusting the white field value and the black field value of the color scale through experiments to obtain a set of parameters that can maximize the accuracy of circle detection.

[0060] Step S2202, using a first search method to search for a first center position from the adjustment result, and using a second search method to search for a second center position from the adjustment result;

[0061] Among them, the first search method is to use the random sampling consistency method to search, and the first center position is the contour shape of the main area of ​​the fisheye image. The second search method is to use the interface in opencv to search, and the second center position is a closed circle in the fisheye image.

[0062] Step S2204, comparing the first distance and the second distance to obtain a comparison result, wherein the first distance is the distance between the first center position and the center position of the corresponding picture, and the second distance is the distance between the second center position and the center position of the corresponding picture;

[0063] Step S2206, obtaining the target graphic corresponding to each image based on the comparison result.

[0064] Optionally, when the first distance is less than the second distance, the first center position is determined as the target graphic corresponding to the image. For example, when the contour shape found by the first search method is closer to the center position of the corresponding image, the contour shape is used as the final detected circle.

[0065] Optionally, when the first distance is greater than the second distance, the second center position is determined as the target graphic corresponding to the image. For example, when the closed circle found by the second search method is closer to the center position of the corresponding image, the closed circle is used as the final detected circle.

[0066] Based on the above steps S2200 to S2206, the color scale of each of the multiple pictures can be adjusted, and then the first center position can be found from the adjustment result using the first search method, and the second center position can be found from the adjustment result using the second search method. Finally, based on the comparison result of the first distance and the second distance, the target graphic corresponding to each picture is obtained, thereby realizing circle detection for multiple pictures.

[0067] As an optional implementation, in step S26, the image expansion results are stitched in a spherical stitching manner to generate a target panoramic image, including:

[0068] Step S260, using the image expansion result, determining the overlapping portion between each expanded image and the remaining expanded images, and performing a cropping process on the overlapping portion to obtain a cropping result;

[0069] In order to estimate the rotation matrix more accurately, when the panoramic camera component rotates 360° to collect multiple pictures, the overlapping area between two adjacent pictures is large, which results in the leftmost and rightmost areas of the equidistant rectangular projection results not being perfectly connected after stitching.

[0070] Optionally, when cropping the overlapping part between each expanded image and the other expanded images, the width of the area to be cropped is determined by adjusting parameters experimentally. In the cropping result, the overlap between adjacent expanded images is reduced, and there is only one seam between adjacent expanded images. The expanded image with the seam separated is cropped into two images, and their upper left corner projection points are calculated respectively to facilitate subsequent cropping and image fusion.

[0071] Step S262, searching for the seams between adjacent expanded images in the cropping result;

[0072] Specifically, the Graph Cut Seam Finder algorithm in OpenCV is used to find the seam between two adjacent expanded images.

[0073] Step S264, stitching the stitching seams using a spherical stitching method to generate an initial panoramic image;

[0074] Specifically, in the above stitching process, multi-band fusion can be used to perform image fusion to generate an initial panoramic image.

[0075] Step S266, performing cropping processing on the area to be cropped in the initial panoramic image to obtain a target panoramic image.

[0076] The above-mentioned area to be cropped is the black area at the upper and lower ends of the initial panoramic image, that is, there are black edges. The black area indicates the area not captured by the camera acquisition component. In order to make the final panoramic image beautiful, the black edges need to be cropped.

[0077] Specifically, the implementation process of performing cropping processing on the to-be-cropped area in the initial panoramic image to obtain the target panoramic image can refer to the introduction of the following embodiment, which will not be described in detail.

[0078] As an optional implementation, in step S266, the area to be cropped in the initial panoramic image is cropped to obtain a target panoramic image including:

[0079] Step S2660, obtaining a mask corresponding to the initial panoramic image;

[0080] Step S2662, searching for a first height and a second height in a height direction of the initial panoramic image based on the mask;

[0081] The first height is the minimum height of the upper black border, and the second height is the maximum height of the lower black border.

[0082] Specifically, based on the mask, the minimum height of the upper black border and the maximum height of the lower black border are found by traversing in the height direction of the initial panorama. Since the traversal operation is time-consuming, the initial panorama after image fusion is divided into small images for traversal operation input, which can effectively reduce the calculation time.

[0083] Step S2664, determining the area to be cropped using the first height and the second height;

[0084] The area to be cropped includes the upper black border and the lower black border. The coordinate origin of the initial panorama is in the upper left corner, the right is the positive direction of the x-axis, and the downward is the positive direction of the y-axis. Therefore, in the y-axis direction, the upper black border is below the minimum height in the initial panorama, and the lower black border is above the maximum height.

[0085] Step S2666, performing cropping processing on the area to be cropped to obtain a target panoramic image.

[0086] Based on the above steps S2660 to S2666, the black edges in the initial panorama can be accurately cropped to avoid mistakenly cropping black objects in the image as black edges, which can further improve the visual effect of the panorama.

[0087] The following is an introduction to the method for generating a panoramic image provided by an embodiment of the present disclosure with reference to a schematic diagram.

[0088] Figure 3 is a schematic diagram of a process of generating a panoramic image according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:

[0089] Step S301, performing a picture reading operation;

[0090] The pictures used for the image reading operation are multiple pictures collected by the panoramic camera component. Figure 4 is a schematic diagram of an input picture according to an embodiment of the present disclosure, such as Figure 4 As shown, 8 fisheye pictures are selected from multiple pictures collected by the panoramic camera component to generate a panoramic picture. The compression format of the fisheye pictures is JPG format, and the size of each picture is 1536*2048.

[0091] Step S302, performing circle detection;

[0092] Specifically, the circle detection implementation process may refer to the above step S220 and will not be described in detail.

[0093] Step S303, performing feature detection;

[0094] Specifically, Scale-Invariant Feature Transform (SIFT) features are used for feature detection and feature descriptor extraction.

[0095] Step S304, performing camera pose estimation based on the features, and obtaining estimation parameter information corresponding to the multiple images;

[0096] Optionally, in step S304, the step of estimating the camera pose based on the feature and obtaining the estimated parameter information corresponding to the multiple images further includes the following steps:

[0097] Step S3041, performing feature matching;

[0098] Step S3042, projecting the feature points onto a spherical surface;

[0099] Step S3043, estimating the internal and external parameters of the camera based on the spherical projection point coordinates;

[0100] Step S3044, optimizing the internal and external parameters of the camera;

[0101] Step S305, determining whether the feature-based camera pose estimation is successful;

[0102] Specifically, if the bundle adjustment optimization algorithm has converged and the estimated parameter values ​​are within a reasonable range, the pose estimation is considered successful, otherwise it is considered a failure.

[0103] Step S306, when it is determined that the feature-based camera pose estimation is unsuccessful, estimating camera internal and external parameters based on gyroscope data;

[0104] Among them, the gyroscope data includes pitch angle, yaw angle and roll angle.

[0105] Step S307, when it is determined that the feature-based camera pose estimation is successful, performing spherical projection and deformation operations;

[0106] Specifically, a spherical projection method is used to expand multiple images to obtain image expansion results.

[0107] Step S308, cutting out the overlapping area;

[0108] Specifically, the implementation process of cutting the overlapping area may refer to the above step S26 and will not be described in detail.

[0109] Step S309, performing image fusion operation;

[0110] Step S310, performing a black border cutting operation;

[0111] For example, Figure 5This is a schematic diagram of a black border of a picture according to an embodiment of the present disclosure. Specifically, the implementation process of cropping the black border operation can refer to the above step S266 and will not be described in detail.

[0112] Step S311, generating a panoramic image.

[0113] For example, Figure 6 A panoramic image is generated according to an embodiment of the present disclosure.

[0114] Based on the above steps S301 to S311, fisheye images with large distortion can be stitched together to obtain a better panoramic stitching effect. At the same time, due to the large viewing angle of fisheye images, fewer input images are required to stitch a scene graph of a scene, and the number of images that the algorithm needs to process is also smaller, so the efficiency of stitching to generate a panoramic image is higher.

[0115] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0117] In the present disclosure, there is also provided a device for generating a panoramic image, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0118] Figure 7 is a structural block diagram of a device for generating a panoramic image according to one embodiment of the present disclosure, such as Figure 7 As shown, a device 700 for generating a panoramic image includes:

[0119] The acquisition module 701 is used to acquire multiple pictures using a panoramic camera assembly;

[0120] An acquisition module 702 is used to acquire estimated parameter information corresponding to a plurality of pictures, wherein the estimated parameter information is used to estimate an expansion effect and a position of an expansion image corresponding to each of the plurality of pictures;

[0121] The expansion module 703 is used to expand the multiple images by using a spherical projection method to obtain the image expansion results;

[0122] The generating module 704 is used to stitch the image expansion results by using a spherical stitching method to generate a target panoramic image.

[0123] Optionally, the acquisition module 702 is used to obtain the estimated parameter information corresponding to multiple pictures, including: performing graphic detection on each of the multiple pictures to obtain a target graphic corresponding to each picture; performing feature detection on the target graphic to extract multiple feature points, and performing pose estimation based on the multiple feature points to determine the estimated parameter information, or determining the estimated parameter information using angular motion detection data collected by the angular motion detection component, wherein the angular motion detection data is detection data recorded synchronously when multiple pictures are collected using a panoramic camera component.

[0124] Optionally, the estimated parameter information includes: panoramic camera component model parameters and a rotation matrix, the panoramic camera component model parameters are used to estimate the expansion effect of the expanded image corresponding to each of the multiple images, and the rotation matrix is ​​used to estimate the position of the expanded image in the target panoramic image.

[0125] Optionally, the acquisition module 702 is also used to perform graphic detection on each of the multiple pictures, and obtaining the target graphic corresponding to each picture includes: adjusting the brightness and contrast of each of the multiple pictures using color scale adjustment to obtain an adjustment result; using a first search method to find a first center position from the adjustment result, and using a second search method to find a second center position from the adjustment result; comparing the first distance with the second distance to obtain a comparison result, wherein the first distance is the distance between the first center position and the center position of the corresponding picture, and the second distance is the distance between the second center position and the center position of the corresponding picture; and obtaining the target graphic corresponding to each picture based on the comparison result.

[0126] Optionally, the generation module 704 is used to stitch the image expansion results using a spherical stitching method, and generating a target panoramic image includes: using the image expansion results to determine the overlapping parts between each expanded image and the remaining expanded images, and cropping the overlapping parts to obtain a cropping result; finding the stitching seams between adjacent expanded images in the cropping result; stitching the stitching seams using a spherical stitching method to generate an initial panoramic image; cropping the area to be cropped in the initial panoramic image to obtain a target panoramic image.

[0127] Optionally, the generation module 704 is also used to crop the area to be cropped in the initial panorama to obtain the target panorama, including: obtaining a mask corresponding to the initial panorama; searching for a first height and a second height in the height direction of the initial panorama based on the mask; determining the area to be cropped using the first height and the second height; and cropping the area to be cropped to obtain the target panorama.

[0128] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, including a memory and at least one processor, wherein the memory stores computer instructions, and the processor is configured to run the computer instructions to execute the steps in any one of the above method embodiments.

[0130] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0131] Optionally, in the present disclosure, the processor may be configured to execute the following steps through a computer program:

[0132] S1, using a panoramic camera component to collect multiple pictures;

[0133] S2, obtaining estimated parameter information corresponding to the multiple images, wherein the estimated parameter information is used to estimate the expansion effect of the expansion image corresponding to each of the multiple images and the position information of the expansion image;

[0134] S3, using spherical projection to expand the multiple images to obtain image expansion results;

[0135] S4, uses spherical stitching to stitch the image expansion results to generate a target panoramic image.

[0136] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are configured to execute the steps of any of the above method embodiments when running.

[0138] Optionally, in this embodiment, the non-transitory computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0139] S1, using a panoramic camera component to collect multiple pictures;

[0140] S2, obtaining estimated parameter information corresponding to the multiple images, wherein the estimated parameter information is used to estimate the expansion effect of the expansion image corresponding to each of the multiple images and the position information of the expansion image;

[0141] S3, using spherical projection to expand the multiple images to obtain image expansion results;

[0142] S4, uses spherical stitching to stitch the image expansion results to generate a target panoramic image.

[0143] Optionally, in this embodiment, the non-transitory computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0144] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product. The program code for implementing the audio processing method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the above embodiments of the present disclosure, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0146] In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0148] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0150] The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present disclosure. These improvements and modifications should also be regarded as the scope of protection of the present disclosure.

Claims

1. A method for generating a panoramic image, comprising: Use panoramic camera components to collect multiple pictures; Adjusting the brightness and contrast of each of the multiple images by using color scale adjustment to obtain an adjustment result; A first search method is used to search for a first center position from the adjustment result, and a second search method is used to search for a second center position from the adjustment result, wherein the first search method is used to represent a random sampling consistency method, and the first center position is used to represent the contour shape of the main area of ​​each of the multiple pictures, and the second search method is used to represent an interface search method in a graphics processing application, and the second center position is used to represent a closed circle position of each of the multiple pictures; Comparing the first distance and the second distance to obtain a comparison result, wherein the first distance is the distance between the first center position and the center position of the corresponding picture, and the second distance is the distance between the second center position and the center position of the corresponding picture; In response to determining that the first distance is smaller than the second distance based on the comparison result, determining a target graphic according to the first center position, wherein the target graphic is used to represent a circle detection result corresponding to each of the multiple pictures; in response to determining that the first distance is larger than the second distance based on the comparison result, determining the target graphic according to the second center position; Performing feature detection on the target graphic to extract multiple feature points, and performing pose estimation based on the multiple feature points to determine estimated parameter information, or determining the estimated parameter information using angular motion detection data collected by an angular motion detection component, wherein the estimated parameter information is used to estimate the expansion effect of the expanded image corresponding to each of the multiple images and the position information of the expanded image, and the angular motion detection data is detection data synchronously recorded when the panoramic camera component is used to collect the multiple images; The plurality of images are expanded by using a spherical projection method to obtain an image expansion result; The image expansion results are stitched using a spherical stitching method to generate a target panoramic image.

2. The method according to claim 1, wherein: The estimated parameter information includes: panoramic camera component model parameters and a rotation matrix, the panoramic camera component model parameters are used to estimate the expansion effect of the expanded image corresponding to each of the multiple images, and the rotation matrix is ​​used to estimate the position of the expanded image in the target panoramic image.

3. The method according to claim 1, wherein: Using the spherical stitching method to stitch the image expansion results to generate the target panoramic image includes: Determine the overlapping portion between each expanded image and the remaining expanded images using the image expansion result, and perform cropping processing on the overlapping portion to obtain a cropping result; Finding the seams between adjacent expanded images in the cropping result; The spherical surface stitching method is used to stitch the stitching seams to generate an initial panoramic image; The target panoramic image is obtained by performing a cropping process on the area to be cropped in the initial panoramic image.

4. The method according to claim 3, wherein: Performing cropping processing on the to-be-cropped area in the initial panoramic image to obtain the target panoramic image includes: Obtaining a mask corresponding to the initial panoramic image; Searching for a first height and a second height in the height direction of the initial panoramic image based on the mask, wherein the first height is used to represent the minimum height of the black edge at the upper end of the initial panoramic image, and the second height is used to represent the maximum height of the black edge at the lower end of the initial panoramic image; Determine the area to be cropped using the first height and the second height; The area to be cropped is cropped to obtain the target panoramic image.

5. A device for generating a panoramic image, comprising: A collection module, used to collect multiple pictures using a panoramic camera component; An acquisition module is used to: adjust the brightness and contrast of each of the multiple pictures by using color scale adjustment to obtain an adjustment result; use a first search method to find a first center position from the adjustment result, and use a second search method to find a second center position from the adjustment result, wherein the first search method is used to represent a random sampling consistency method, the first center position is used to represent the contour shape of the main area of ​​each of the multiple pictures, the second search method is used to represent an interface search method in a graphics processing application, and the second center position is used to represent the closed circular position of each of the multiple pictures; compare the first distance with the second distance to obtain a comparison result, wherein the first distance is the distance between the first center position and the center position of the corresponding picture, and the second distance is the distance between the second center position and the center position of the corresponding picture ; In response to determining that the first distance is less than the second distance based on the comparison result, a target graphic is determined according to the first center position, wherein the target graphic is used to represent the circle detection result corresponding to each of the multiple pictures; in response to determining that the first distance is greater than the second distance based on the comparison result, the target graphic is determined according to the second center position; performing feature detection on the target graphic to extract multiple feature points, and performing pose estimation based on the multiple feature points to determine estimated parameter information, or determining the estimated parameter information using angular motion detection data collected by an angular motion detection component, wherein the estimated parameter information is used to estimate the expansion effect of the expanded image corresponding to each of the multiple pictures and the position of the expanded image, and the angular motion detection data is detection data synchronously recorded when the panoramic camera component is used to collect the multiple pictures; An expansion module, used for expanding the plurality of images by using a spherical projection method to obtain an image expansion result; The generation module is used to stitch the image expansion results using a spherical stitching method to generate a target panoramic image.

6. An electronic device comprising: Panoramic camera assembly; Angular motion detection component; at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.

8. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images