Panoramic image synthesis device, panoramic camera and image synthesis method

By designing image preprocessing, synthesis and postprocessing modules in a panoramic camera, combining an inertial measurement unit and a graphics processor, the problem of inaccurate image noise and color generation of panoramic cameras is solved, and efficient and accurate panoramic image synthesis is achieved.

CN118413733BActive Publication Date: 2024-11-29LABPANO TECHNOLOGY (CHANGZHOU) CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410883000.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-03
Publication Date
2024-11-29
Estimated Expiration
2044-07-03

AI Technical Summary

Technical Problem

When generating panoramic images, due to the limitations of the embedded operating system, the generated image noise is high, the colors are inaccurate, and the image data cannot be transmitted in time when it cannot communicate with the terminal, which affects the speed and quality of image synthesis.

Method used

A panoramic image synthesis device is designed, including an image preprocessing module, an image synthesis module and an image postprocessing module. By performing color and resolution processing on the fisheye image, jitter correction is performed in combination with the motion state information of the inertial measurement unit, and using The graphics processor performs image fusion and stitching to ensure image accuracy and speed.

Benefits of technology

Improves the accuracy and generation speed of panoramic images, reduces noise and color interference, and can quickly generate high-quality panoramic images without relying on the terminal.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118413733B_ABST
    Figure CN118413733B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a panoramic image synthesis device, a panoramic camera and an image synthesis method. The device comprises: an image pre-processing module connected to multiple fisheye lenses in a panoramic camera respectively, used to obtain multiple frames of fisheye images taken by each fisheye lens, and perform a first image preprocessing on the multiple frames of fisheye images; an image synthesis module connected to multiple fisheye lenses and the image pre-processing module in a panoramic camera respectively, used to send pre-configured shooting parameters to the fisheye lenses, obtain multiple frames of fisheye images after the first image preprocessing by the image pre-processing module, perform panoramic image synthesis based on the multiple frames of fisheye images after the first image preprocessing, and obtain a panoramic image; an image post-processing module connected to the image synthesis module, used to receive the panoramic image transmitted by the image synthesis module, and output the panoramic image after performing a second image preprocessing. The use of this device can reduce the influence of environmental factors and accurately generate a panoramic image in a panoramic camera.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of panoramic cameras, and in particular to a panoramic image synthesis device, a panoramic camera and an image synthesis method. Background Art

[0002] With the improvement of people's living standards and the change of consumption concepts, users' demand for high-quality, multi-angle shooting is increasing. Panoramic cameras can take 360-degree panoramic photos and videos with unique visual effects, which can meet the personalized needs of consumers. Due to the power consumption and real-time requirements of current panoramic cameras, the internal operating system is usually an embedded operating system such as Linux.

[0003] In traditional technology, due to the development limitations of embedded operating systems such as Linux, most traditional panoramic cameras directly transmit the captured image data to various terminals for panoramic image synthesis, thereby obtaining a more accurate panoramic image. In some application scenarios, users need to generate panoramic images in a timely and fast manner. Traditional technology also requires the use of additional terminals for panoramic image synthesis, which is slow in generating panoramic images, and the synthesis effect of panoramic images will also be affected by terminal performance. Affected by various factors, when the panoramic camera cannot communicate with the terminal, the user cannot transmit the image data to various terminals for image processing. If the panoramic image is synthesized using an embedded operating system such as Linux inside the panoramic camera, the generated panoramic image will have more noise and poor color, and an accurate panoramic image cannot be obtained. Summary of the invention

[0004] Based on this, it is necessary to provide a panoramic image synthesis device, a panoramic camera and an image synthesis method that reduce the influence of environmental factors and can accurately generate panoramic images in a panoramic camera to address the above technical problems.

[0005] In a first aspect, the present disclosure provides a panoramic image synthesis device. The panoramic image synthesis device is arranged in a panoramic camera, and the device comprises:

[0006] An image pre-processing module, connected to the multiple fisheye lenses in the panoramic camera, respectively, for acquiring multiple frames of fisheye images taken by each of the fisheye lenses, and performing a first image pre-processing on the multiple frames of fisheye images, wherein the first image pre-processing at least includes processing the colors and resolutions of the multiple frames of fisheye images;

[0007] an image synthesis module, connected to the multiple fisheye lenses in the panoramic camera and the image pre-processing module respectively, for sending pre-configured shooting parameters to the fisheye lenses, obtaining the multiple frames of fisheye images after the first image pre-processing by the image pre-processing module, and performing panoramic image synthesis based on the multiple frames of fisheye images after the first image pre-processing to obtain a panoramic image;

[0008] The image post-processing module is connected to the image synthesis module, and is used to receive the panoramic image transmitted by the image synthesis module, and output the panoramic image after performing a second image pre-processing on the panoramic image, wherein the second image pre-processing includes processing the imaging effect of the panoramic image.

[0009] In one embodiment, the device further comprises a graphics processor: the graphics processor is connected to the image post-processing module, and the image synthesis module is connected to the inertial measurement unit in the panoramic camera and the graphics processor respectively;

[0010] The image synthesis module is further used to perform frame dropping processing on at least one frame of the multiple frames of fisheye images after the first image preprocessing according to the timestamp of each frame of fisheye image captured by each fisheye lens, and to obtain the motion state information of the panoramic camera measured by the inertial measurement unit, and perform jitter correction on the fisheye image obtained after the frame dropping processing based on the motion state information;

[0011] The image synthesis module is further used to send the fisheye image obtained after the jitter correction to the graphics processor to instruct the graphics processor to perform image fusion and image stitching based on the fisheye image obtained after the jitter correction to obtain a panoramic image.

[0012] In one embodiment, when the panoramic camera includes a graphics processor, the image synthesis module is also connected to the graphics processor in the panoramic camera; the image synthesis module is also used to send only the fisheye image obtained after the jitter correction to the graphics processor in the panoramic camera;

[0013] The graphics processor in the panoramic camera performs image fusion and image stitching based on the fisheye image obtained after the shake correction to obtain a panoramic image, and sends the panoramic image to the image synthesis module.

[0014] In one embodiment, the image synthesis module includes: a frame loss processing module, which is connected to the image pre-processing module, and is used to obtain a plurality of fisheye images taken by each of the fisheye lenses after pre-processing of the first image, determine at least one frame of fisheye image that does not have a matching relationship among the plurality of fisheye images taken by each of the fisheye lenses according to the timestamps of the plurality of fisheye images, delete at least one frame of fisheye image that does not have a matching relationship from the plurality of fisheye images taken by each of the fisheye lenses, and obtain a fisheye image with frame loss processing.

[0015] In one of the embodiments, the device is further provided with a storage unit, which is connected to the image pre-processing module, and the storage unit stores camera image processing parameters pre-burned using OTP. The image pre-processing module is used to perform a first image pre-processing on the multiple frames of fisheye images based on the camera image processing parameters.

[0016] In one of the embodiments, the multiple fisheye lenses include a first fisheye lens and a second fisheye lens, and the image synthesis module is connected to the first fisheye lens and the second fisheye lens, respectively, and is used to control the first fisheye lens and the second fisheye lens to capture fisheye images at the same time.

[0017] In one embodiment, the image pre-processing module includes: a first pre-processing module and a second pre-processing module, the first pre-processing module is connected to the first fisheye lens and the image synthesis module respectively, and the first pre-processing module is used to perform first image pre-processing on multiple frames of fisheye images captured by the first fisheye lens;

[0018] The second pre-processing module is connected to the second fisheye lens and the image synthesis module respectively, and the second pre-processing module is used to perform first image pre-processing on a plurality of frames of fisheye images captured by the second fisheye lens.

[0019] In one of the embodiments, an operating system is provided in the panoramic camera, and the operating system includes a hardware abstraction layer, and the image pre-processing module, the image synthesis module and the image post-processing module are processed in the hardware abstraction layer.

[0020] In one of the embodiments, the panoramic image synthesis device is a single integrated chip.

[0021] In a second aspect, the present disclosure further provides a panoramic camera, the panoramic camera comprising:

[0022] Multiple fisheye lenses, and

[0023] A panoramic image synthesis device as described in any one of the above.

[0024] In a third aspect, the present disclosure further provides an image synthesis method, which is applied to the panoramic image synthesis device in any one of the above embodiments or the panoramic camera in the above embodiments, and the method includes:

[0025] In response to an image synthesis request, sending pre-configured shooting parameters to a plurality of fisheye lenses in a panoramic camera;

[0026] Acquire multiple frames of fisheye images captured by each of the fisheye lenses based on the shooting parameters, and perform first image preprocessing on the multiple frames of fisheye images, wherein the first image preprocessing at least includes processing the colors and resolutions of the multiple frames of fisheye images;

[0027] Performing panoramic image synthesis based on the multiple frames of fisheye images after the first image preprocessing to obtain a panoramic image;

[0028] The panoramic image is subjected to second image preprocessing and then outputted, wherein the second image preprocessing includes processing of an imaging effect of the panoramic image.

[0029] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0030] In the above embodiments, the image pre-processing module is used to perform the first image pre-processing on multiple frames of fisheye images, and the color, resolution, etc. of the multiple frames of fisheye images can be adjusted, thereby ensuring the accuracy of the color of the fisheye image. The higher the resolution adjustment, the more details the fisheye image contains, and the higher the accuracy of the synthesized panoramic image. The lower the resolution adjustment, the faster the panoramic image synthesis can be. The image synthesis module is connected to multiple fisheye lenses in the panoramic camera, and the pre-configured shooting parameters are sent to the fisheye lenses, which can ensure that the parameters used by each fisheye lens during shooting are unified, thereby improving the subsequent panoramic image synthesis and the speed of the first image pre-processing. The image synthesis module can also be connected to the image pre-processing module to perform panoramic image synthesis based on the multiple frames of fisheye images after the first image pre-processing. Since the multiple frames of fisheye images are pre-processed by the first image, the color of the obtained panoramic image will be more accurate. The image post-processing module is connected to the image synthesis module, and is used to receive the panoramic image transmitted by the image synthesis module, and output the panoramic image after performing a second image pre-processing. Since the panoramic image may be subject to interference such as color inaccuracy and noise after synthesis, the panoramic image can also be subjected to a second image pre-processing to obtain a more accurate panoramic image that is more in line with the actual shooting scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 is a schematic structural diagram of a panoramic image synthesis device in one embodiment;

[0033] Figure 2 is a schematic diagram of the structure of a graphics processor in one embodiment;

[0034] Figure 3 is a schematic diagram of the structure between a panoramic image synthesis device and a panoramic camera in one embodiment;

[0035] Figure 4 is a structural schematic diagram of an image synthesis module in one embodiment;

[0036] Figure 5 is a schematic structural diagram of a storage unit in one embodiment;

[0037] Figure 6 is a schematic diagram of the structure of an image pre-processing module in one embodiment;

[0038] Figure 7 A schematic diagram of an operating system architecture in one embodiment;

[0039] Figure 8 FIG. 4 is a flow chart of an image synthesis method in an embodiment.

[0040] The components in the accompanying drawings are marked as follows: 100, panoramic image synthesis device; 120, image pre-processing module; 140, image synthesis module; 160, image post-processing module; 180, graphics processor; 142, frame loss processing module; 144, jitter correction module; 190, storage unit; 122, first pre-processing module; 124, second pre-processing module. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0042] It should be noted that the terms "comprises" and "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or equipment.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0044] In the disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the disclosure can be understood according to the specific circumstances.

[0045] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it may be directly on the other element or there may be an intermediate element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element at the same time.

[0046] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0047] In one embodiment, Figure 1As shown, a panoramic image synthesis device 100 is provided. The panoramic image synthesis device 100 can be set in a panoramic camera. The panoramic camera can include: multiple fisheye lenses, IMU (Inertial Measurement Unit), etc. The panoramic image synthesis device 100 includes: an image pre-processing module 120, an image synthesis module 140 and an image post-processing module 160.

[0048] The image pre-processing module 120 is respectively connected to the multiple fisheye lenses in the panoramic camera, and is used to obtain multiple frames of fisheye images taken by each fisheye lens, and perform a first image preprocessing on the multiple frames of fisheye images, wherein the first image preprocessing at least includes processing the color and resolution of the multiple frames of fisheye images.

[0049] Among them, a panoramic camera is a camera that can take panoramic photos or panoramic videos. Through special lenses and shooting methods, it can capture a wider field of view and completely record the surrounding environment. A panoramic camera can take panoramic photos in the horizontal or vertical direction, or it can take 360-degree panoramic photos or videos by rotating the lens. The special lens in a panoramic camera is usually a fisheye lens, which is a wide-angle lens with a very large viewing angle and distortion effect. Its name comes from the shape of its lens that is similar to a fisheye. The distortion effect of the fisheye lens makes the object in the center of the image appear larger, while the objects at the edge are bent or distorted. In order to ensure that the panoramic image can be accurately synthesized, the panoramic lens mentioned in some embodiments of the present disclosure can usually capture a field of view greater than 180 degrees, and can usually capture a field of view of 220 degrees. A fisheye image is a wide-angle perspective image, which is an image captured by a special lens (fisheye lens). Fisheye images have a strong distortion effect, and objects in the center of the image appear normal size, while objects farther away from the center are stretched and distorted. This special distortion effect makes fisheye images uniquely valuable in some specific application fields, such as panoramic photography, security monitoring, virtual reality, etc. Fisheye images can be corrected through post-processing to convert them into normal perspective images and eliminate the distortion effect. This can make fisheye images more consistent with the visual perception of the human eye and easier to understand and use. The first image preprocessing is usually the initial processing of the fisheye image, such as adjusting the resolution of the fisheye image, adjusting the color of the fisheye image, removing the noise of the fisheye image, etc. The first image preprocessing can be determined based on the panoramic image synthesis requirements to meet the current synthesis scene.

[0050] Specifically, the image pre-processing module 120 can be connected to the image synthesis module 140 and multiple fisheye lenses in the panoramic camera respectively. The image pre-processing module 120 can obtain multiple frames of fisheye images captured by each fisheye lens, and perform a first image pre-processing on each frame of the fisheye image. For example, noise reduction, color correction, etc. are performed on each frame of the fisheye image.

[0051] In some exemplary embodiments, the image pre-processing module 120 may perform a first image pre-processing on multiple frames of fisheye images based on an ISP (Image Signal Processor), for example, it may include: white balance calibration, color calibration, denoising, resolution adjustment, etc. The ISP can automatically adjust the parameters of the fisheye image according to factors such as ambient light and image content to obtain a clearer, more realistic and high-quality image. In addition, according to different application scenarios and application requirements, those skilled in the art can flexibly adjust the resolution. For example, if the current demand is to obtain a panoramic image with a higher resolution and there is no requirement for the speed of synthesizing the panoramic image, the resolution can be adjusted to 4K or 5K. For another example, if the current demand requires a faster output of a panoramic image, and the minimum resolution requirement is 2K, the resolution can be adjusted to 2K, thereby increasing the speed of subsequent panoramic image synthesis.

[0052] The image synthesis module 140 is respectively connected to the multiple fisheye lenses in the panoramic camera and the image pre-processing module 120, and is used to send the pre-configured shooting parameters to the fisheye lenses, and obtain the multiple frames of fisheye images after the first image preprocessing by the image processing module, and perform panoramic image synthesis based on the preprocessed multiple frames of fisheye images to obtain a panoramic image.

[0053] Among them, shooting parameters generally refer to some parameters that can be adjusted and set during the process of photography or video recording to affect the quality and effect of the final image. Shooting parameters may include, for example, the frame rate and focal length of each fisheye lens sampling. Panoramic image synthesis is a process of combining multiple photos taken at different angles or positions into a panoramic image using multiple image stitching technology. This technology can create a very wide field of view, making the viewer feel as if they are in the shooting scene.

[0054] Specifically, the image synthesis module 140 can be connected to multiple fisheye lenses in the panoramic camera and send pre-configured shooting parameters to the multiple fisheye lenses. After each fisheye lens receives the shooting parameters, it can shoot based on the shooting parameters when it needs to shoot a fisheye image, thereby obtaining multiple frames of fisheye images. The image synthesis module 140 can also be connected to the image pre-processing module 120, and can obtain multiple frames of fisheye images after the image pre-processing module 120 has undergone first image preprocessing, and then perform panoramic image synthesis based on the multiple frames of fisheye images after the first image preprocessing, to obtain a panoramic image synthesized from multiple frames of fisheye images.

[0055] The image post-processing module 160 is connected to the image synthesis module 140, and is used to receive the panoramic image transmitted by the image synthesis module 140, and output the panoramic image after performing a second image pre-processing on the panoramic image, wherein the second image pre-processing includes processing the imaging effect of the panoramic image.

[0056] The processing of imaging effects may include: image denoising, sharpening, contrast adjustment, color mapping, etc.

[0057] Specifically, the image post-processing module 160 can be connected to the image synthesis module 140. After the image synthesis module 140 synthesizes a panoramic image, the image synthesis module 140 can send the panoramic image to the image post-processing module 160. After receiving the panoramic image, the image post-processing module 160 performs a second image pre-processing on the panoramic image and outputs it. The image post-processing module 160 can output the panoramic image after the second pre-processing to a panoramic camera, or output the panoramic image after the second pre-processing to a display screen or an encoder. According to different application scenarios, those skilled in the art can flexibly set the position of the image output by the image post-processing module 160. In some embodiments of the present disclosure, the position of the output image is not limited.

[0058] In the above-mentioned panoramic image synthesis device, the image pre-processing module 120 is used to perform the first image pre-processing on multiple frames of fisheye images, and the color, resolution, etc. of the multiple frames of fisheye images can be adjusted, thereby ensuring the accuracy of the color of the fisheye image. The higher the resolution adjustment, the more details the fisheye image contains, and the higher the accuracy of the synthesized panoramic image. The lower the resolution adjustment, the faster the panoramic image synthesis can be. The image synthesis module 140 is connected to multiple fisheye lenses in the panoramic camera, and sends the pre-configured shooting parameters to the fisheye lenses, which can ensure that the parameters used by each fisheye lens during shooting are unified, thereby improving the subsequent panoramic image synthesis and the speed of the first image pre-processing. The image synthesis module 140 can also be connected to the image pre-processing module 120 to perform panoramic image synthesis based on the multiple frames of fisheye images after the first image pre-processing. Since the multiple frames of fisheye images are pre-processed by the first image, the color of the obtained panoramic image will be more accurate. The image post-processing module 160 is connected to the image synthesis module 140, and is used for receiving the panoramic image transmitted by the image synthesis module 140, and outputting the panoramic image after performing a second image pre-processing. Since the panoramic image may be subject to interference such as color inaccuracy and noise after synthesis, the panoramic image may also be subjected to a second image pre-processing, so as to obtain a more accurate panoramic image that is more in line with the actual shooting scene.

[0059] In one embodiment, Figure 2 As shown, the device further includes a graphics processor 180: the graphics processor 180 is connected to the image post-processing module 160, and the image synthesis module 140 is respectively connected to multiple fisheye lenses ( Figure 2 (not shown), the inertial measurement unit in the panoramic camera, and the graphics processor 180 are connected.

[0060] The image processor may be a GPU (graphics processing unit). The inertial measurement unit may be an IMU (Inertial Measurement Unit), which is a device that integrates multiple inertial sensors. IMU usually includes sensors such as accelerometers, gyroscopes, and magnetometers, which are used to measure parameters such as acceleration, angular velocity, and direction of an object, and output the object's motion state information through data fusion technology. In a camera, the IMU can obtain the camera's motion state and posture information.

[0061] The image synthesis module 140 is further used to perform frame dropping processing on at least one frame of the multiple frames of fisheye images after preprocessing the first image according to the timestamp of each frame of fisheye image taken by each fisheye lens, and to obtain the motion state information of the panoramic camera measured by the inertial measurement unit, and to perform jitter correction on the fisheye image obtained after the frame dropping processing based on the motion state information.

[0062] Among them, the frame drop processing in some embodiments of the present disclosure may be a processing method in which at least one frame of fisheye image that does not meet the requirements is discarded and no subsequent panoramic image synthesis is performed. The motion state information may include: three-axis acceleration of the camera, three-axis angular velocity data, etc. Jitter correction is a technology used to process the jitter phenomenon that appears in the image, aiming to reduce or eliminate image blur or distortion problems caused by camera shake, handheld shooting or other factors. Jitter refers to irregular shaking or vibration in the image, which may cause blurred image details, chaotic edges or unclear overall image. In order to correct this jitter, image jitter correction technology usually uses an image stabilization algorithm to process the image to make the image visually clearer and more stable. In some embodiments of the present disclosure, the timestamp may usually be the generation time of the fish image.

[0063] Specifically, the image synthesis module 140 can also determine the timestamp of each frame of fisheye image obtained by taking each fisheye image, and then match the timestamp of each frame of fisheye image obtained by taking each fisheye image, and group the fisheye images with timestamps close to those taken by different fisheye images, and discard the fisheye images that cannot be grouped without subsequent processing.

[0064] In addition, after the image synthesis module 140 is connected to the inertial measurement unit, the image synthesis module 140 can also obtain the IMU information sequence in the inertial measurement unit, and the IMU information sequence can include motion information of the panoramic camera, such as three-axis acceleration, three-axis angular velocity data, etc. The image synthesis module 140 can use the running status information to perform jitter correction on each frame of the fisheye image obtained after the frame loss processing.

[0065] The image synthesis module 140 is further used to send only the fisheye image obtained after the jitter correction to the graphics processor 180, so as to instruct the graphics processor 180 to perform image fusion and image stitching based on the fisheye image obtained after the jitter correction to obtain a panoramic image.

[0066] Image fusion refers to the process of combining multiple fisheye images taken by different fisheye cameras to produce a panoramic image. In image fusion, different images or different parts of images can be combined, fused, and superimposed to achieve some specific visual effects or purposes. Image stitching is the process of stitching multiple local images or partial images into a complete large image. In image stitching, multiple images are usually aligned and stitched according to certain rules to form a seamless overall image.

[0067] Specifically, in order to improve the efficiency of panoramic image synthesis, the processing of the panoramic image can be executed in the graphics processor 180. The image synthesis module 140 can send the fisheye image obtained after jitter correction to the graphics processor 180, and the graphics processor 180 can perform image fusion and image stitching based on the fisheye image obtained after jitter correction to obtain a panoramic image. In addition, the interaction between the image synthesis module 140 and the graphics processor can be achieved by sharing memory. It should be noted that image fusion and image stitching can be executed simultaneously or in a preset order. In some embodiments of the present disclosure, the execution order of image fusion and image stitching is not limited.

[0068] In this embodiment, by performing frame loss processing on multiple fisheye images, fisheye images that do not meet the requirements can be discarded, thereby ensuring the accuracy of subsequent panoramic image synthesis. In addition, performing jitter correction on the fisheye images obtained after the frame loss processing can further ensure the accuracy of panoramic image synthesis. In addition, the image fusion and image stitching involved in the panoramic image synthesis process are executed in the graphics processing, which can fully utilize the performance of the graphics processor and improve the efficiency of panoramic image synthesis.

[0069] In one embodiment, Figure 3 As shown, when the panoramic camera includes a graphics processor, the image synthesis module 140 is also connected to the graphics processor in the panoramic camera. The image synthesis module 140 is also used to send only the fisheye image obtained after the jitter correction to the graphics processor in the panoramic camera;

[0070] The graphics processor in the panoramic camera performs image fusion and image stitching based on the fisheye image obtained after the shake correction to obtain a panoramic image, and sends the panoramic image to the image synthesis module 140 .

[0071] Specifically, when the panoramic camera also contains a graphics processor, the image synthesis module 140 will not send the fisheye image obtained after the shake correction to the graphics processor in the panoramic image synthesis device 100. The image synthesis module 140 will only send the fisheye image obtained after the shake correction to the graphics processor in the panoramic camera. The graphics processor in the panoramic camera performs image fusion and image stitching to obtain a panoramic image. For how to perform image fusion and image stitching here, please refer to the above embodiment, and will not be repeated here. After the graphics processor in the panoramic camera synthesizes the panoramic image, the panoramic image can be sent to the image synthesis module 140.

[0072] In this embodiment, when the panoramic camera also contains a graphics processor, the image synthesis module 140 is only sent to the graphics processor in the panoramic camera. The graphics processor in the panoramic camera is used for processing, which can reduce the resource consumption in the panoramic image synthesis device and make full use of the resources in the panoramic camera, thereby improving the efficiency of panoramic image synthesis.

[0073] In one embodiment, Figure 4 As shown, the image synthesis module 140 includes: a frame loss processing module 142 and a jitter correction module 144. The frame loss processing module 142 is connected to the image pre-processing module 120, and is used to obtain multiple frames of fisheye images taken by each of the fisheye lenses after the first image pre-processing, determine at least one frame of fisheye image that does not have a matching relationship among the multiple frames of fisheye images taken by each of the fisheye lenses according to the timestamps of the multiple frames of fisheye images, delete at least one frame of fisheye image that does not have a matching relationship from the multiple frames of fisheye images taken by each of the fisheye lenses, and obtain the fisheye image with frame loss processing.

[0074] Among them, the matching relationship can usually be used to determine the correlation relationship between fisheye images taken by different fisheye lenses. When the fisheye images taken by different fisheye lenses are correlated, for example, they are taken at the same time, it can usually be proved that the fisheye images have a matching relationship. When using fisheye images with matching relationships for image fusion and image stitching, a more accurate panoramic image can be synthesized.

[0075] Specifically, the image synthesis module 140 may include a frame loss processing module 142 and a jitter correction module 144. The frame loss processing module 142 may be connected to the image pre-processing module 120. The frame loss processing module 142 may obtain multiple frames of fisheye images captured by each fisheye lens after the first image pre-processing by the image pre-processing module 120, and determine the timestamp of each frame of fisheye image. The multiple frames of fisheye images captured by each fisheye lens are matched and grouped using the timestamp, and the fisheye frame images with similar or identical timestamps are grouped together. The fisheye images grouped together are determined to have a matching relationship, and the fisheye images that are not grouped are determined to have no matching relationship, and then at least one frame of fisheye image that has no matching relationship is determined. The at least one frame of fisheye image that has no matching relationship is deleted, so as to obtain the fisheye image after the frame loss processing.

[0076] In an exemplary embodiment, two fisheye lenses A and fisheye lens B are used as examples for explanation. Fisheye lens A captures 4 frames of fisheye images, namely a1, a2, a3 and a4, and fisheye lens B only captures 3 frames of fisheye images, namely b1, b2 and b3. Among them, the timestamps of a1 and b1 are close, the timestamps of a2 and b2 are close, and the timestamps of a4 and b3 are close. Then a1 and b1 can be divided into a group, and a matching relationship is determined between a1 and b1, a matching relationship is determined between a2 and b2, and a matching relationship is determined between a4 and b3. If a3 does not have a matching relationship, it can be determined that a3 is a fisheye image that does not have a matching relationship. a3 can be deleted.

[0077] In another embodiment, if fisheye lenses A and B both capture 2 frames of fisheye images, where fisheye lens A captures two frames of fisheye images a1 and a2, and fisheye lens B captures two frames of fisheye images b1 and b2. The timestamp of the a1 fisheye image is close to the timestamp of the b2 fisheye image, it can be determined that there is a matching relationship between a1 and b2, while the timestamp of a2 is not close to the timestamp of b1 and the timestamp of b2, and the timestamp of b1 is not close to the timestamp of a1 and a2, it can be determined that a2 and b1 are fisheye images that do not have a matching relationship.

[0078] The jitter correction module 144 is connected to the frame loss processing module 142 and the graphics processor, and is used to send the fisheye image obtained after jitter correction to the graphics processor. It should be noted that the graphics processor mentioned here can be a graphics processor in a panoramic camera or a graphics processor in an image synthesis device.

[0079] Specifically, the jitter correction module 144 can use the three-axis acceleration, three-axis angular velocity data and other information of the panoramic camera, combined with filtering algorithm models such as Kalman filtering, to solve the panoramic camera posture parameters of the next frame. The algorithm then uses the camera posture parameters to rotate the fisheye image of the next frame, thereby achieving jitter correction for each frame of the fisheye image.

[0080] In addition, it should be noted that after the matching relationship is determined, fisheye images with the matching relationship can be used for image stitching and image fusion in subsequent image stitching and image fusion, which can improve processing efficiency.

[0081] In this embodiment, the matching relationship between fisheye images taken by different fisheye lenses can be determined through the timestamp, and the fisheye images without a matching relationship are discarded, which can ensure the accuracy of subsequent panoramic image synthesis and improve the efficiency of panoramic image synthesis.

[0082] In one embodiment, Figure 5As shown, the device is also provided with a storage unit 190, which can be a register or an EEPROM (Electrically Erasable Programmable read only memory). The storage unit 190 is connected to the image pre-processing module 120. The storage unit 190 pre-stores the camera image processing parameters burned by OTP. The image pre-processing module 120 is used to perform a first image pre-processing on the multiple frames of fisheye images based on the camera image processing parameters.

[0083] Among them, OTP burning (One-Time Programmable programming) is used to program certain fixed data or keys into the chip at one time. The role of OTP burning is to write critical configuration information, encryption keys or other fixed data into the chip, and once written, these data cannot be modified or erased. Camera image processing parameters can include white balance parameters, exposure time parameters, resolution parameters, autofocus parameters, etc.

[0084] Specifically, since the imaging range of the fisheye lens is relatively wide, the two fisheye lenses are likely to have very different lighting environments when shooting, resulting in large differences in the fisheye images obtained after imaging by the two fisheye lenses. Therefore, in order to reduce the differences between the fisheye images taken by different fisheye lenses in the panoramic camera, thereby improving the effect of panoramic image synthesis, the camera image processing parameters can be pre-burned into the storage unit 190 using OTP, and the camera image processing parameters in the subsequent storage unit 190 cannot be modified. Therefore, the camera image processing parameters can be used to uniformly adjust the fisheye images taken by different fisheye lenses.

[0085] In addition, it should be noted that, for a panoramic camera with a storage unit, the camera image processing parameters can be written into the storage unit of the panoramic camera by OTP burning to release additional storage space of the panoramic image synthesis device 100 for storing the camera processing parameters.

[0086] In this embodiment, the camera image processing parameters are written into the storage unit 190 by OTP burning. Since the data after OTP burning cannot be changed, the camera image processing parameters stored in the storage unit 190 can be used to accurately and uniformly adjust the fisheye image.

[0087] In one embodiment, the plurality of fisheye lenses include: a first fisheye lens and a second fisheye lens. The image synthesis module 140 is connected to the first fisheye lens and the second fisheye lens respectively, and is used to control the first fisheye lens and the second fisheye lens to capture fisheye images at the same time.

[0088] Specifically, under normal circumstances, the range of the fisheye lens shooting is about 180°, so two fisheye lenses can cover a panoramic view of about 360°. In addition, due to the presence of two fisheye lenses, the shooting time of the fisheye lenses needs to be synchronized to ensure the accuracy of the subsequent fisheye images when performing panoramic image synthesis. If they are not synchronized, the panoramic image obtained after synthesis usually has a poor match. For example, fisheye lens A shoots at 10s and fisheye lens B shoots at 20s. Between 10s and 20s, the environment changes, and various problems may occur in the final synthesized panoramic image. Therefore, in order to avoid differences in the shooting time of the first fisheye lens and the second fisheye lens, the image synthesis module 140 can also control the first fisheye lens and the second fisheye lens to shoot fisheye images at the same time. That is, the hardware synthesis module can control the first fisheye lens and the second fisheye lens to achieve hard synchronization. Hard synchronization generally refers to a method of ensuring synchronization performance by using physical connection or hardware signal transmission between systems or devices. Hard synchronization can achieve synchronization between devices through clock signals, trigger signals or other physical connection methods provided by hardware devices, so that they operate according to precise time standards or trigger conditions.

[0089] In some exemplary embodiments, hard synchronization between the first fisheye lens and the second fisheye lens may be achieved by any of the following methods:

[0090] Use hardware synchronization signal: Many camera devices provide external trigger interfaces, and the start acquisition time of the two fisheye lenses can be synchronized through external trigger signals. For example, a dedicated synchronization signal generator can be integrated in the image synthesis module 140 to send an external synchronization signal to achieve hard synchronization. Or the first fisheye lens is used as the master and the second fisheye lens is used as the slave, and hard synchronization is achieved by connecting an external synchronization signal.

[0091] The image synthesis module 140 may also be directly connected to multiple fisheye lenses and ensure that they always start collecting images at the same time point.

[0092] In this embodiment, the image synthesis module 140 is used to control the first fisheye lens and the second fisheye lens to capture fisheye images at the same time, which can ensure the accuracy of subsequent panoramic image synthesis.

[0093] In one embodiment, Figure 6 As shown, the image pre-processing module 120 includes: a first pre-processing module 122 and a second pre-processing module 124, the first pre-processing module 122 is connected to the first fisheye lens and the image synthesis module 140 respectively, and the first pre-processing module 122 is used to perform first image pre-processing on the multiple frames of fisheye images taken by the first fisheye lens;

[0094] The second pre-processing module 124 is connected to the second fisheye lens and the image synthesis module 140 respectively. The second pre-processing module 124 is used to perform first image pre-processing on the multiple frames of fisheye images captured by the second fisheye lens.

[0095] In this embodiment, the fisheye image taken by the first fisheye lens is processed by the first pre-processing module 122, and the fisheye image taken by the second fisheye lens is processed by the second pre-processing module 124. The fisheye images taken by different fisheye lenses can be processed separately, and there will be no interference between different fisheye images. This ensures that in the subsequent synthesis process, it is possible to accurately determine which fisheye images are taken by the first fisheye lens and which fisheye images are taken by the second fisheye lens. Therefore, the fisheye images taken by the first fisheye lens and the fisheye images taken by the second fisheye lens are synthesized, which can improve the accuracy of panoramic image synthesis.

[0096] In one embodiment, an operating system is provided in the panoramic camera, and the operating system includes a hardware abstraction layer. The image pre-processing module 120, the image synthesis module 140 and the image post-processing module 160 are processed in the hardware abstraction layer.

[0097] The operating system may be an Android system, in which the HAL layer refers to the Hardware Abstraction Layer. The HAL layer of the Android system provides a series of standard interfaces, so that the upper layer system of Android (such as application framework or system service) can communicate with the underlying hardware device without knowing the specific hardware details.

[0098] Specifically, Figure 7As shown, the operating system may include: an application layer, an architecture layer, and a hardware abstraction layer. In conventional technology, an additional terminal is usually used to synthesize panoramic images. Therefore, in the process of panoramic image synthesis, it is all completed at the application layer. Due to the influence of various factors such as user experience, terminal compatibility, and terminal performance, when the panoramic image is synthesized at the application layer, the synthesis effect of the panoramic image will be affected to a certain extent. Usually, only panoramic images with a resolution of 4K and a frame rate of 30 can be processed at the application layer. In order to ensure the synthesis effect of the panoramic image, the image pre-processing module 120, the image synthesis module 140, and the image post-processing module 160 can be processed in the hardware abstraction layer. Since the image pre-processing module 120, the image synthesis module 140, and the image post-processing module 160 are processed in the hardware abstraction layer, the functional characteristics of the hardware in the image pre-processing module 120, the image synthesis module 140, and the image post-processing module 160 can be directly called, and technologies such as hardware acceleration can be used to process higher resolution videos. Therefore, the image pre-processing module 120, the image synthesis module 140 and the image post-processing module 160 will have more powerful processing capabilities and performance advantages closer to hardware when processed in the hardware abstraction layer, and can support processing of higher resolution images, such as 5.7K resolution and 30 frame rate images.

[0099] In this embodiment, the image pre-processing module 120, the image synthesis module 140 and the image post-processing module 160 are processed in the hardware abstraction layer, which can improve the performance of panoramic image synthesis and obtain a panoramic image with higher resolution.

[0100] In one embodiment, the panoramic image synthesis device 100 is a single integrated chip, wherein the single integrated chip may integrate the functions of each module in the panoramic image synthesis device 100 mentioned in the above embodiment.

[0101] Each module in the panoramic image synthesis device 100 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a microprocessor in the form of hardware, or can be stored in the memory of a laser radar device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0102] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or implemented as hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the present embodiment.

[0103] In one embodiment, the present disclosure further provides a panoramic camera, the panoramic camera comprising:

[0104] Multiple fisheye lenses;

[0105] And, the panoramic image synthesis device 100 in any one of the above embodiments.

[0106] In one embodiment, the present disclosure further provides an image synthesis method, which is applied to the panoramic image synthesis device 100 or the panoramic camera mentioned in any of the above embodiments. Figure 8 As shown, the method includes:

[0107] S802, in response to an image synthesis request, sending pre-configured shooting parameters to a plurality of fisheye lenses in a panoramic camera.

[0108] The image synthesis request may generally be a request for capturing a panoramic image, which may be achieved by pressing a button in the panoramic camera or by issuing an instruction through software.

[0109] S804, obtaining a plurality of fisheye images captured by each of the fisheye lenses based on the shooting parameters, and performing a first image preprocessing on the plurality of fisheye images, wherein the first image preprocessing at least includes processing the colors and resolutions of the plurality of fisheye images.

[0110] S806, performing panoramic image synthesis based on the multiple frames of fisheye images after the first image preprocessing to obtain a panoramic image.

[0111] S808, performing second image preprocessing on the panoramic image and then outputting the image, where the second image preprocessing includes processing an imaging effect of the panoramic image.

[0112] In one embodiment, the panoramic camera further includes an inertial measurement unit, and the device further includes: a graphics processor; the panoramic image synthesis is performed on the multiple frames of fisheye images after preprocessing the first image to obtain a panoramic image, including:

[0113] performing frame dropping processing on at least one frame of the multiple frames of fisheye images after the first image preprocessing according to the timestamp of each frame of the fisheye image captured by each of the fisheye lenses;

[0114] Acquire motion state information of the panoramic camera measured by the inertial measurement unit, and perform jitter correction on the fisheye image obtained after the frame loss processing based on the motion state information;

[0115] The fisheye image obtained after the jitter correction is sent to the graphics processor to instruct the graphics processor to perform image fusion and image stitching based on the fisheye image obtained after the jitter correction to obtain a panoramic image.

[0116] In one embodiment, when the panoramic camera includes a graphics processor, the method further includes:

[0117] Only the fisheye image obtained after the jitter correction is sent to the graphics processor in the panoramic camera to instruct the graphics processor in the panoramic camera to perform image fusion and image stitching based on the fisheye image obtained after the jitter correction to obtain a panoramic image, and send the panoramic image to the image synthesis module.

[0118] In one embodiment, the performing frame dropping processing on at least one frame of the multiple frames of fisheye images after the first image preprocessing according to the timestamp of each frame of the fisheye image captured by each of the fisheye lenses includes:

[0119] Acquire multiple frames of fisheye images captured by each of the fisheye lenses after preprocessing the first image;

[0120] Determine, according to the timestamps of the multiple frames of fisheye images, at least one frame of fisheye image for which there is no matching relationship among the multiple frames of fisheye images captured by each of the fisheye lenses;

[0121] At least one frame of fisheye image that does not have a matching relationship is deleted from the multiple frames of fisheye images captured by each of the fisheye lenses to obtain a fisheye image with frame loss processing.

[0122] In one embodiment, the panoramic image synthesis device is further provided with a storage unit, wherein the storage unit stores the camera image processing parameters pre-burned using OTP. The first image preprocessing of the multiple fisheye images includes:

[0123] Based on the camera image processing parameters, a first image preprocessing is performed on the multiple frames of fisheye images.

[0124] In one embodiment, the plurality of fisheye lenses include a first fisheye lens and a second fisheye lens, and the method further includes:

[0125] The first fisheye lens and the second fisheye lens are controlled to capture fisheye images at the same time.

[0126] In one embodiment, the panoramic camera is provided with an operating system, the operating system includes a hardware abstraction layer, and the method further includes:

[0127] The method is applied in the hardware abstraction.

[0128] For the specific implementation and limitation of this embodiment, please refer to the above embodiment, which will not be repeated here.

[0129] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0130] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to this.

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A panoramic image synthesis device, characterized in that: The panoramic image synthesis device is arranged in a panoramic camera, and the device comprises: An image pre-processing module, connected to the multiple fisheye lenses in the panoramic camera, respectively, for acquiring multiple frames of fisheye images taken by each of the fisheye lenses, and performing a first image pre-processing on the multiple frames of fisheye images, wherein the first image pre-processing at least includes processing the colors and resolutions of the multiple frames of fisheye images; an image synthesis module, connected to the multiple fisheye lenses in the panoramic camera and the image pre-processing module respectively, for sending pre-configured shooting parameters to the fisheye lenses, obtaining the multiple frames of fisheye images after the first image pre-processing by the image pre-processing module, and performing panoramic image synthesis based on the multiple frames of fisheye images after the first image pre-processing to obtain a panoramic image; The image synthesis module is connected to the inertial measurement unit in the panoramic camera, and is further used to perform frame loss processing on at least one frame of the multiple frames of fisheye images after the first image preprocessing according to the timestamp of each frame of fisheye image captured by each fisheye lens, and obtain the motion state information of the panoramic camera measured by the inertial measurement unit, and perform jitter correction on the fisheye image obtained after the frame loss processing based on the motion state information; The image post-processing module is connected to the image synthesis module, and is used to receive the panoramic image transmitted by the image synthesis module, and output the panoramic image after performing a second image pre-processing on the panoramic image, wherein the second image pre-processing includes processing the imaging effect of the panoramic image.

2. The device according to claim 1, characterized in that The device further comprises a graphics processor: the graphics processor is connected to the image post-processing module, and the image synthesis module is also connected to the graphics processor; The image synthesis module is further used to send the fisheye image obtained after the jitter correction to the graphics processor to instruct the graphics processor to perform image fusion and image stitching based on the fisheye image obtained after the jitter correction to obtain a panoramic image.

3. The device according to claim 2, characterized in that When the panoramic camera includes a graphics processor, the image synthesis module is also connected to the graphics processor in the panoramic camera; the image synthesis module is also used to send only the fisheye image obtained after the jitter correction to the graphics processor in the panoramic camera; The graphics processor in the panoramic camera performs image fusion and image stitching based on the fisheye image obtained after the shake correction to obtain a panoramic image, and sends the panoramic image to the image synthesis module.

4. The device according to claim 1, characterized in that The image synthesis module includes: a frame loss processing module, which is connected to the image pre-processing module and is used to obtain multiple frames of fisheye images taken by each of the fisheye lenses after pre-processing of the first image, determine at least one frame of fisheye image that does not have a matching relationship among the multiple frames of fisheye images taken by each of the fisheye lenses according to the timestamps of the multiple frames of fisheye images, delete at least one frame of fisheye image that does not have a matching relationship from the multiple frames of fisheye images taken by each of the fisheye lenses, and obtain the fisheye image with frame loss processing.

5. The device according to any one of claims 1 to 4, characterized in that The device is also provided with a storage unit, which is connected to the image pre-processing module. The storage unit stores camera image processing parameters pre-burned using OTP. The image pre-processing module is used to perform first image pre-processing on the multiple frames of fisheye images based on the camera image processing parameters.

6. The device according to any one of claims 1 to 4, characterized in that The multiple fisheye lenses include a first fisheye lens and a second fisheye lens. The image synthesis module is connected to the first fisheye lens and the second fisheye lens, respectively, and is used to control the first fisheye lens and the second fisheye lens to capture fisheye images at the same time.

7. The device according to claim 6, characterized in that The image pre-processing module includes: a first pre-processing module and a second pre-processing module, wherein the first pre-processing module is connected to the first fisheye lens and the image synthesis module respectively, and the first pre-processing module is used to perform first image pre-processing on the multiple frames of fisheye images captured by the first fisheye lens; The second pre-processing module is connected to the second fisheye lens and the image synthesis module respectively, and the second pre-processing module is used to perform first image pre-processing on a plurality of frames of fisheye images captured by the second fisheye lens.

8. The device according to any one of claims 1 to 4, characterized in that The panoramic camera is provided with an operating system, which includes a hardware abstraction layer. The image pre-processing module, the image synthesis module and the image post-processing module are processed in the hardware abstraction layer.

9. The device according to any one of claims 1 to 4, characterized in that The panoramic image synthesis device is a single integrated chip.

10. A panoramic camera, characterized in that: The panoramic camera comprises: Multiple fisheye lenses, and The panoramic image synthesis device according to any one of claims 1 to 9.

11. An image synthesis method, characterized in that: Applied to the panoramic image synthesis device according to any one of claims 1 to 9 or the panoramic camera according to claim 10, the method comprises: In response to an image synthesis request, sending pre-configured shooting parameters to a plurality of fisheye lenses in a panoramic camera; Acquire multiple frames of fisheye images captured by each of the fisheye lenses based on the shooting parameters, and perform first image preprocessing on the multiple frames of fisheye images, wherein the first image preprocessing at least includes processing the colors and resolutions of the multiple frames of fisheye images; Performing panoramic image synthesis based on the multiple frames of fisheye images after the first image preprocessing to obtain a panoramic image; The panoramic image is subjected to second image preprocessing and then outputted, wherein the second image preprocessing includes processing of an imaging effect of the panoramic image.

Citation Information

Patent Citations

  • In-camera generation of high quality composite panoramic images

    CN101867720A

  • Positional sensor-assisted image registration for panoramic photography

    CN103534726A

  • Dynamic object tracking method and system based on fisheye lens

    CN107730438A

  • Panoramic camera, and image generation method, system, device and storage medium thereof

    CN107995421A

  • Mobile terminal image synthesis method, mobile terminal image synthesis apparatus and mobile terminal

    US20190068877A1