A panoramic image generation method, device, equipment and storage medium

Abstract

The present application discloses a panoramic image generation method, device, equipment and storage medium, and relates to the field of image processing technology. The method comprises: respectively acquiring regional images acquired by at least two image acquisition devices; mapping the regional images to the target imaging sphere to obtain regional spherical images; based on the HS optical flow algorithm, layering the regional spherical images to obtain at least one layered spherical image; according to the layered spherical images, fusing at least two regional spherical images to obtain a panoramic image. The above technical scheme, by layering the regional spherical images and stitching and fusing the panoramic images in layers, solves the problem of ghosting and misalignment in the panoramic stitching picture under large parallax, and ensures the naturalness and smoothness of the panoramic stitching picture.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, in particular to the field of image processing technology, and specifically to a panoramic image generation method, device, equipment and storage medium. Background Art

[0002] With the development of information technology, 360-degree VR panorama has begun to enter people's daily lives.

[0003] The target 360-degree VR panorama is stitched and fused in scenes with small parallax. In scenes with small parallax, multiple pinhole / wide-angle / fisheye cameras are approximately at the same spatial position. However, in scenes with large parallax, the distribution positions of multiple cameras in space are different. When observing the same physical target from different cameras, different images and distortions will be presented due to different viewing angles and occlusion conditions.

[0004] Therefore, using traditional VR panoramic stitching and fusion methods in scenes with large parallax will cause the stitched images to have very obvious ghosting and misalignment. Summary of the invention

[0005] The present application provides a panoramic image generation method, device, equipment and storage medium to ensure the naturalness and smoothness of the panoramic stitching picture.

[0006] According to one aspect of the present application, a method for generating a panoramic image is provided, the method comprising:

[0007] Respectively acquiring regional images acquired by at least two image acquisition devices;

[0008] Mapping the regional image to a target imaging sphere to obtain a regional spherical image;

[0009] Based on the HS optical flow algorithm, the regional spherical image is layered to obtain at least one layered spherical image;

[0010] According to the layered spherical image, at least two regional spherical images are fused to obtain a panoramic image.

[0011] According to another aspect of the present application, a panoramic image generation device is provided, the device comprising:

[0012] An image acquisition module, used to respectively acquire regional images acquired by at least two image acquisition devices;

[0013] A spherical image determination module, used for mapping the regional image to a target imaging sphere to obtain a regional spherical image;

[0014] An image layering module, used for layering the regional spherical image based on the HS optical flow algorithm to obtain at least one layered spherical image;

[0015] The image fusion module is used to fuse at least two regional spherical images according to the layered spherical image to obtain a panoramic image.

[0016] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement any panoramic image generation method provided in the embodiments of the present application.

[0020] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any panoramic image generation method provided in the embodiments of the present application is implemented.

[0021] The present application obtains regional images acquired by at least two image acquisition devices respectively; maps the regional images to the target imaging sphere to obtain regional spherical images; based on the HS optical flow algorithm, the regional spherical images are layered to obtain at least one layered spherical image; according to the layered spherical images, at least two regional spherical images are fused to obtain a panoramic image. The above technical solution solves the problem of ghosting and misalignment of panoramic stitching images under large parallax by layering the regional spherical images and stitching and fusing the panoramic images in layers, thereby ensuring the naturalness and smoothness of the panoramic stitching images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flowchart of a panoramic image generation method provided according to Embodiment 1 of the present application;

[0023] Figure 2 is a flow chart of a panoramic image generation method provided according to Embodiment 2 of the present application;

[0024] Figure 3 is a structural schematic diagram of a panoramic image generating device provided according to Embodiment 3 of the present application;

[0025] Figure 4 It is a structural schematic diagram of an electronic device that implements the panoramic image generation method of an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In addition, it should be noted that in the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of relevant data such as regional images, regional spherical images and hierarchical features involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0029] Embodiment 1

[0030] Figure 1 This is a flowchart of a panoramic image generation method provided according to the first embodiment of the present application. This embodiment is applicable to the case of stitching and fusing 360-degree VR panoramic images, especially to the case of stitching and fusing 360-degree VR panoramic images in large parallax scenes. It can be executed by a panoramic image generation device, which can be implemented in the form of hardware and / or software. The panoramic image generation device can be configured in a computer device, such as an image acquisition device. Figure 1 As shown, the method includes:

[0031] S110, respectively obtain regional images acquired by at least two image acquisition devices.

[0032] The image acquisition device refers to a device used to capture, record and store images, and may be at least one of a digital camera or a mobile phone camera, etc. The regional image refers to an image captured by the image acquisition device from the current position or a specified area.

[0033] Specifically, at least two image acquisition devices are installed under the target scene, and the devices are ensured to be stable and fixed in an appropriate manner; and the image acquisition devices are used to shoot or capture the required regional images. The target scene refers to the scene that needs to be captured with a panoramic image.

[0034] It should be noted that the position of the image acquisition device must be determined to ensure that each image acquisition device can capture 360-degree image content of the target scene.

[0035] Optionally, the image acquisition device used in the embodiment of the present application may be a pinhole camera or a wide-angle camera.

[0036] It is understandable that the use of a pinhole camera can give the image an infinite depth of field, ensuring that the subject remains clear in both the front and back planes; the use of an optical angle camera can capture more details and environment at close distances.

[0037] S120 , mapping the regional image to the target imaging sphere to obtain a regional spherical image.

[0038] The target imaging sphere refers to a pre-established imaging sphere for projecting the regional image. Optionally, the target imaging sphere is an imaging sphere in a world coordinate system. The regional curved surface image refers to an image of the regional image mapped onto the target imaging sphere.

[0039] Specifically, based on the camera imaging model, a mapping relationship between pixel points on the regional image and target imaging spherical coordinates is established, and according to the mapping relationship, the regional image is mapped to the target imaging sphere to obtain a regional spherical image.

[0040] It should be noted that the camera imaging model is a mathematical model used to describe how a camera converts a three-dimensional object into a two-dimensional image in an optical system, and can be at least one of a pinhole camera model and a camera coordinate system.

[0041] Optionally, the target imaging sphere can be determined by the following steps:

[0042] Step 1: Define the center of the imaging sphere, take the center as the origin, and establish a rectangular coordinate system.

[0043] Step 2: Suppose there are n image acquisition devices, and the coordinates of the n image acquisition devices in the rectangular coordinate system are (x i ,y i , z i ), i∈[1,n].

[0044] Wherein, n refers to the number of image acquisition devices, and i is the label of each image acquisition device.

[0045] Step 3: satisfy the formula min∑ in ＝1 x i 2 +y i 2 +z i 2 The coordinate origin of is determined as the target coordinate origin.

[0046] Step 4: Determine the target imaging sphere according to the target coordinate origin and the radius of the target imaging sphere.

[0047] Among them, the radius of the target imaging sphere can be set manually according to actual conditions or empirical values, or determined through a large number of experiments; generally, the radius setting range of the target imaging sphere is 20 meters to 50 meters.

[0048] S130 . Based on the HS optical flow algorithm, the regional spherical image is layered to obtain at least one layered spherical image.

[0049] Among them, the HS (Horn-Schunch) optical flow algorithm is an algorithm used to calculate the motion pattern of pixels in a video sequence to obtain the optical flow field between consecutive frames; optical flow refers to the pixel motion pattern observed between consecutive frames, and the motion information of the object is obtained by analyzing the movement of pixels in the image. The layered spherical image refers to the spherical image obtained by layering the regional spherical image.

[0050] Specifically, the HS optical flow algorithm is used to layer the regional spherical image according to a preset speed rule to obtain at least one layered spherical image.

[0051] It should be noted that the preset speed rule refers to the rule that the near view moves faster in the video and the far view moves slower in the video; and the layered regional spherical image also represents the separation of the near view image and the far view image in the regional spherical image.

[0052] S140: According to the layered spherical image, fuse at least two regional spherical images to obtain a panoramic image.

[0053] Specifically, the layered spherical images are screened, and the layered spherical images of the same level in the layered spherical images are fused to obtain a panoramic image.

[0054] Among them, the level is used to characterize the field distance of the layered spherical image; the higher the level, the longer the field distance, the smaller the impact of parallax on panoramic stitching, and the better the stitching fusion; the lower the level, the smaller the field distance, and the greater the impact of parallax on panoramic stitching.

[0055] It should be noted that layered spherical images at different levels cannot be fused.

[0056] The embodiment of the present application obtains regional images acquired by at least two image acquisition devices respectively; maps the regional images to the target imaging sphere to obtain regional spherical images; based on the HS optical flow algorithm, the regional spherical images are layered to obtain at least one layered spherical image; according to the layered spherical images, at least two regional spherical images are fused to obtain a panoramic image. The above technical solution solves the problem of ghosting and misalignment of panoramic stitching images under large parallax by layering the regional spherical images and stitching and fusing the panoramic images in layers, thereby ensuring the naturalness and smoothness of the panoramic stitching images.

[0057] Embodiment 2

[0058] Figure 2 This is a flow chart of a panoramic image generation method provided in accordance with the second embodiment of the present application. Based on the technical solutions of the above embodiments, this embodiment refines "according to the layered spherical images, at least two regional spherical images are fused to obtain a panoramic image" into "feature extraction of the layered spherical images to obtain the hierarchical features of the layered spherical images; according to the hierarchical features, at least two regional spherical images are fused to obtain a panoramic image". It should be noted that for the parts not described in detail in the embodiments of the present application, please refer to the relevant descriptions of other embodiments. Figure 2 As shown, the method includes:

[0059] S210, respectively obtain regional images acquired by at least two image acquisition devices.

[0060] S220 , mapping the regional image to the target imaging sphere to obtain a regional spherical image.

[0061] S230 , based on the HS optical flow algorithm, the regional spherical image is layered to obtain at least one layered spherical image.

[0062] S240 , extracting features from the layered spherical image to obtain hierarchical features of the layered spherical image.

[0063] Among them, hierarchical features refer to features in layered spherical images, which can be represented in matrix or vector form; optionally, it can be scale-invariant feature transform (SIFT), which is used to characterize feature points of scale invariance, rotation invariance, illumination invariance and local properties in the image; scale invariance means that it can be detected and described at different scales, and even if the size of the target object in the image changes, the corresponding key points can still be extracted; rotation invariance means that it can remain stable when the image is rotated, and similar feature points can still be extracted even if the direction of the object changes; illumination invariance means that it has a certain robustness to illumination changes, and similar feature points can still be extracted even if the brightness or contrast of the image changes; local properties refer to features based on local areas of the image, which can represent local texture, edge and other feature information in the image, and have the ability to distinguish to a certain extent.

[0064] Specifically, based on the feature extraction algorithm, feature extraction is performed on feature points in the layered spherical image to obtain hierarchical features of the layered spherical image.

[0065] It should be noted that the feature extraction algorithm may be a Speeded-Up Scale-Invariant Feature Transform (sSIFT) algorithm, which is not specifically limited in the embodiments of the present application.

[0066] Exemplarily, the sSIFT algorithm is used to extract features of the layered spherical image to obtain spherical sSIFT feature points, and the sSIFT feature points are used as hierarchical features of the layered spherical image.

[0067] Furthermore, based on the scene motion speed, a near-view spherical image is determined from at least one layered spherical image; and features of the near-view spherical image are extracted to obtain layered features of the near-view spherical image.

[0068] The scene motion speed refers to the speed of the scene in the video stream; the near scene moves fast in the video, and the far scene moves slowly in the video. The near scene spherical image refers to a spherical image whose scene motion speed is not less than a preset speed threshold; the preset speed threshold can be set manually according to actual conditions or experience, or determined through a large number of experiments.

[0069] Specifically, based on the scene motion speed and a preset speed threshold, a spherical image whose scene motion speed is not less than the preset speed threshold is screened out from at least one layered spherical image as a close-up spherical image; and feature extraction is performed on feature points in the close-up spherical image based on a feature extraction algorithm to obtain hierarchical features of the close-up spherical image.

[0070] It is understandable that the distant scene moves slowly in the video stream and has a small parallax, which will not cause ghosting and misalignment problems in the panoramic stitching picture. Therefore, the distant spherical images can be directly stitched and fused, and only the features of the near spherical images are extracted, which improves the image stitching efficiency.

[0071] S250: Fusing at least two regional spherical images according to the hierarchical features to obtain a panoramic image.

[0072] The panoramic image refers to an image obtained by mapping, stitching and fusing regional images.

[0073] Specifically, according to the hierarchical features, the same hierarchical features in at least two layered spherical images of the same level are determined, and at least two same hierarchical features of the same level are spliced ​​and fused, that is, the regional spherical images corresponding to the at least two layered spherical images are spliced ​​and fused to obtain a panoramic image.

[0074] Optionally, based on the hierarchical features, the overlapping area between the spherical images of adjacent regions is determined; based on the overlapping area, the best projection sphere and / or the best stitching line is determined; based on the best projection sphere and / or the best stitching line, at least two regional spherical images are fused to obtain a panoramic image.

[0075] The overlapped area refers to the overlapping area between the spherical images of adjacent areas. The best projection sphere refers to the best projection position of the object in the target imaging sphere. The best stitching line refers to the best stitching position of the overlapped area between the spherical images of adjacent areas.

[0076] Specifically, according to the hierarchical features, the areas where there are overlapping feature points between the spherical images of adjacent regions are screened out as the overlapping areas; according to the overlapping areas, the best projection sphere and / or the best stitching line are determined; according to the best projection sphere and / or the best stitching line, at least two regional spherical images are fused to obtain a panoramic image.

[0077] It can be understood that by introducing the best projection sphere and / or the best stitching line to perform image fusion, it helps to ensure that the imaging surface is natural and smooth.

[0078] Optionally, a first overlapping spherical area and a second overlapping spherical area are determined based on the overlapping area; and under the premise of ensuring normal imaging, an optimal projection sphere is determined based on a first distance from the scene to the first overlapping spherical area and a second distance from the scene to the second overlapping spherical area.

[0079] The first overlapping spherical area refers to the corresponding overlapping area in the first regional spherical image of two adjacent regional spherical images. The second overlapping spherical area refers to the corresponding overlapping area in the second regional spherical image of two adjacent regional spherical images. The first distance refers to the physical distance between the first overlapping area and the scene. The second distance refers to the physical distance between the second overlapping area and the scene.

[0080] Specifically, the corresponding overlapping areas in the spherical images of adjacent areas are determined according to the overlapping areas between the spherical images of adjacent areas; and under the premise of ensuring normal imaging, based on the distance matching algorithm, the best projection sphere is determined according to the first distance from the scene to the first overlapping spherical area and the second distance from the scene to the second overlapping spherical area.

[0081] It should be noted that the distance matching algorithm comprehensively considers factors such as application requirements, camera position and angle, field of view and focal length, and visualization requirements, and is selected based on specific circumstances.

[0082] Optionally, a first overlapping spherical area and a second overlapping spherical area are determined according to the overlapping area; and an optimal stitching line of the overlapping area is determined according to pixel information in the first overlapping spherical area and pixel information in the second overlapping spherical area.

[0083] The pixel information refers to the pixel points in the spherical area; the pixel point is the smallest visible element in the image; each pixel point represents a minimum area in the image, and its position is determined by the horizontal coordinate and the vertical coordinate.

[0084] Specifically, according to the overlapped area between the spherical images of the adjacent areas, the corresponding overlapped areas in the spherical images of the adjacent areas are determined; and based on the neighborhood calculation method, according to the pixel information in the first overlapped spherical area and the pixel information in the second overlapped spherical area, the neighborhood means corresponding to the first overlapped spherical area and the second overlapped spherical area are determined, and the neighborhood means are subtracted to obtain the target neighborhood absolute value; according to the target neighborhood absolute value, the minimum value is selected in a row or column manner, and then the best stitching line of the overlapped area is determined. Alternatively, for each pixel point in the first overlapped spherical area, the neighborhood of the pixel point is determined, the neighborhood corresponding to the pixel point is determined from the second overlapped spherical area, the corresponding pixel points in the two neighborhoods are subtracted, and the pixel point with the smallest absolute difference value is selected as the point on the best stitching line; and so on, all the pixel points in the first overlapped spherical area are traversed to obtain all the pixel points with the smallest absolute difference value, and all the pixel points with the smallest absolute difference value are connected to obtain the best stitching line.

[0085] It should be noted that the neighborhood calculation method can be at least one of four-neighborhood calculation or eight-neighborhood calculation, and the embodiment of the present application does not specifically limit this.

[0086] The embodiment of the present application obtains regional images acquired by at least two image acquisition devices respectively; maps the regional images to the target imaging sphere to obtain regional spherical images; based on the HS optical flow algorithm, the regional spherical images are layered to obtain at least one layered spherical image; features are extracted from the layered spherical images to obtain hierarchical features of the layered spherical images; and at least two regional spherical images are fused according to the hierarchical features to obtain a panoramic image. The above technical solution, by extracting features from the layered spherical images and fusing images based on the hierarchical features, solves the problems of ghosting and misalignment in panoramic stitching images under large parallax, while ensuring the flexibility and accuracy of hierarchical image stitching.

[0087] Embodiment 3

[0088] Figure 3 This is a schematic diagram of the structure of a panoramic image generation device provided in accordance with the third embodiment of the present application, which can be applied to the case of stitching and fusing 360-degree VR panoramic images, especially to the case of stitching and fusing 360-degree VR panoramic images in large parallax scenes. The panoramic image generation device can be implemented in the form of hardware and / or software, and the panoramic image generation device can be configured in a computer device, such as an image acquisition device. Figure 3 As shown, the device comprises:

[0089] An image acquisition module 310 is used to respectively acquire regional images acquired by at least two image acquisition devices;

[0090] A spherical image determination module 320 is used to map the regional image to the target imaging sphere to obtain a regional spherical image;

[0091] An image layering module 330, configured to layer the regional spherical image based on the HS optical flow algorithm to obtain at least one layered spherical image;

[0092] The image fusion module 340 is used to fuse at least two regional spherical images according to the layered spherical images to obtain a panoramic image.

[0093] The embodiment of the present application obtains regional images acquired by at least two image acquisition devices respectively; maps the regional images to the target imaging sphere to obtain regional spherical images; based on the HS optical flow algorithm, the regional spherical images are layered to obtain at least one layered spherical image; according to the layered spherical images, at least two regional spherical images are fused to obtain a panoramic image. The above technical solution solves the problem of ghosting and misalignment of panoramic stitching images under large parallax by layering the regional spherical images and stitching and fusing the panoramic images in layers, thereby ensuring the naturalness and smoothness of the panoramic stitching images.

[0094] Optionally, the image fusion module 340 includes:

[0095] A first feature determination unit is used to extract features from the layered spherical image to obtain hierarchical features of the layered spherical image;

[0096] The image fusion unit is used to fuse at least two regional spherical images according to the hierarchical features to obtain a panoramic image.

[0097] Optionally, the image fusion module 340 further includes:

[0098] An image determining unit, configured to determine a near-view spherical image from at least one layered spherical image based on a scene motion speed;

[0099] A second feature determination unit is used to extract features from the close-up spherical image to obtain hierarchical features of the close-up spherical image;

[0100] The image fusion unit is used to fuse at least two regional spherical images according to the hierarchical features to obtain a panoramic image.

[0101] Optionally, the image fusion unit includes:

[0102] A region determination subunit, used to determine the overlapped region between the spherical images of adjacent regions according to the hierarchical features;

[0103] A surface line determination subunit, used to determine the best projection sphere and / or the best suture line according to the overlapped area;

[0104] The image fusion subunit is used to fuse at least two regional spherical images according to the best projection sphere and / or the best stitching line to obtain a panoramic image.

[0105] Optionally, the surface line determination subunit is specifically used for:

[0106] Determine a first overlapping spherical surface area and a second overlapping spherical surface area according to the overlapping area;

[0107] Under the premise of ensuring normal imaging, the best projection spherical surface is determined according to a first distance from the scene to the first coincident spherical surface area and a second distance from the scene to the second coincident spherical surface area.

[0108] Optionally, the surface line determination subunit is further specifically used for:

[0109] Determine a first overlapping spherical surface area and a second overlapping spherical surface area according to the overlapping area;

[0110] An optimal stitching line of the overlapped area is determined according to the pixel information in the first overlapped spherical area and the pixel information in the second overlapped spherical area.

[0111] Optionally, the image acquisition device is a pinhole camera or a wide-angle camera.

[0112] The panoramic image generation device provided in the embodiments of the present application can execute the panoramic image generation method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects for executing each panoramic image generation method.

[0113] Embodiment 4

[0114] Figure 4 4 is a schematic diagram of the structure of an electronic device 410 that implements the panoramic image generation method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0115] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In RAM413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, ROM412 and RAM413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0116] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 performs the various methods and processes described above, such as a panoramic image generation method.

[0118] In some embodiments, the panoramic image generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the panoramic image generation method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured as a panoramic image generation method in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0126] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for generating a panoramic image, characterized in that: include: Respectively acquiring regional images acquired by at least two image acquisition devices; Mapping the regional image to a target imaging sphere to obtain a regional spherical image; Based on the HS optical flow algorithm, the regional spherical image is layered to obtain at least one layered spherical image; According to the layered spherical image, at least two regional spherical images are fused to obtain a panoramic image.

2. The method according to claim 1, characterized in that The step of fusing at least two regional spherical images according to the layered spherical images to obtain a panoramic image comprises: Performing feature extraction on the layered spherical image to obtain hierarchical features of the layered spherical image; According to the hierarchical features, at least two regional spherical images are fused to obtain a panoramic image.

3. The method according to claim 1, characterized in that The step of fusing at least two regional spherical images according to the layered spherical images to obtain a panoramic image comprises: Determining a near-view spherical image from the at least one layered spherical image based on a scene motion speed; Extracting features of the close-up spherical image to obtain hierarchical features of the close-up spherical image; According to the hierarchical features, at least two regional spherical images are fused to obtain a panoramic image.

4. The method according to claim 2 or 3, characterized in that: The step of fusing at least two regional spherical images according to the hierarchical features to obtain a panoramic image includes: According to the hierarchical features, the overlapping areas between the spherical images of adjacent areas are determined; Determine the best projection sphere and / or the best suture line according to the overlapped area; At least two regional spherical images are fused according to the optimal projection spherical surface and / or the optimal stitching line to obtain a panoramic image.

5. The method according to claim 4, characterized in that Determining the best projection sphere according to the overlapped area includes: Determine a first overlapping spherical surface area and a second overlapping spherical surface area according to the overlapping area; Under the premise of ensuring normal imaging, the best projection spherical surface is determined according to a first distance from the scene to the first overlapped spherical surface area and a second distance from the scene to the second overlapped spherical surface area.

6. The method according to claim 4, characterized in that According to the overlapped area, determining the best suture line includes: Determine a first overlapping spherical surface area and a second overlapping spherical surface area according to the overlapping area; An optimal stitching line of the overlapped area is determined according to the pixel information in the first overlapped spherical area and the pixel information in the second overlapped spherical area.

7. The method according to claim 1, characterized in that The image acquisition device is a pinhole camera or a wide-angle camera.

8. A panoramic image generation device, characterized in that: include: An image acquisition module, used to respectively acquire regional images acquired by at least two image acquisition devices; A spherical image determination module, used for mapping the regional image to a target imaging sphere to obtain a regional spherical image; An image layering module, used for layering the regional spherical image based on the HS optical flow algorithm to obtain at least one layered spherical image; The image fusion module is used to fuse at least two regional spherical images according to the layered spherical image to obtain a panoramic image.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic image generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the panoramic image generation method as described in any one of claims 1 to 7 is implemented.

Images