Panoramic image processing method and panoramic camera

Abstract

Embodiments of the present invention disclose a panoramic image processing method and a panoramic camera, which are used to improve the anti-shake effect of the panoramic camera. The method of the embodiments of the present invention includes: obtaining a panoramic image, the acquisition timestamp of the pixels of the panoramic image, and a plurality of rotation matrices. Converting the panoramic image into a spherical image, and for each pixel of the spherical image, respectively performing position correction using the plurality of rotation matrices to obtain a plurality of corrected pixels for each pixel. For each pixel of the spherical image, according to the acquisition timestamp of the pixels of the spherical image, determining a first corrected pixel and a second corrected pixel from the plurality of corrected pixels. For each pixel of the spherical image, performing weighted average calculation using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel. Through the panoramic image processing method of the embodiments of the present invention, each pixel of the spherical image can be corrected to the position where the pixel should be at the initial moment as much as possible, thereby achieving a good anti-shake effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a panoramic image processing method and a panoramic camera. Background Art

[0002] A panoramic camera is used to collect panoramic images. Specifically, a plurality of lenses are provided on the panoramic camera, and each lens can be used to collect a fisheye image. By stitching the fisheye images collected by each lens on the panoramic camera, a panoramic image can be obtained.

[0003] When the panoramic camera collects a panoramic image, the panoramic camera may shake, resulting in deformation of the collected fisheye image. Since multiple fisheye images need to be stitched, after stitching the deformed fisheye images, the imaging effect of the obtained panoramic image is poor. Therefore, it is necessary to improve the anti-shake effect of the panoramic camera. Summary of the Invention

[0004] Embodiments of the present invention provide a panoramic image processing method and a panoramic camera for improving the anti-shake effect of the panoramic camera.

[0005] To achieve this purpose, the embodiments of the present invention adopt the following technical solutions:

[0006] A panoramic image processing method includes:

[0007] Obtain a panoramic image and the acquisition timestamp of the pixels of the panoramic image;

[0008] Obtain a plurality of rotation matrices, where the panoramic image is obtained by the panoramic camera during a target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period;

[0009] Convert the panoramic image into a spherical image, and the acquisition timestamps of the mutually matching pixels on the panoramic image and the spherical image are the same;

[0010] For each pixel of the spherical image, use the plurality of rotation matrices to perform position correction respectively to obtain a plurality of corrected pixels for each pixel. The positions of the different corrected pixels belonging to the same pixel are the positions where the pixels on the spherical image are moved according to the poses represented by the different rotation matrices;

[0011] For each pixel of the spherical image, a first corrected pixel and a second corrected pixel are determined from a plurality of the corrected pixels according to the acquisition timestamp of the pixel of the spherical image, wherein the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel;

[0012] For each pixel of the spherical image, weighted average calculation is performed using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, wherein in the weighted average calculation, the weight magnitude relationship between the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship between the first duration and the second duration, the first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix.

[0013] Optionally, a plurality of CMOS (Complementary Metal Oxide Semiconductor) sensors are provided on the panoramic camera, and one CMOS sensor is used to acquire one fisheye image, and the positioning timestamp is the moment when a row of photosensitive pixels in the CMOS sensor is scanned;

[0014] The obtaining of the panoramic image and the acquisition timestamp of the pixel of the panoramic image includes:

[0015] Images are synchronously acquired by a plurality of the CMOS sensors during the target period to obtain a plurality of the fisheye images;

[0016] The plurality of fisheye images are stitched to obtain a panoramic image;

[0017] The acquisition timestamp of the pixel of the panoramic image is obtained;

[0018] After the weighted average calculation is performed on each pixel of the spherical image using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, the panoramic image processing method further includes:

[0019] Each target corrected pixel of the spherical image is projected onto a plane to obtain a corrected panoramic image.

[0020] Optionally, the plurality of rotation matrices are respectively a first row rotation matrix, a middle rotation matrix, and a last row rotation matrix;

[0021] The positioning timestamp of the first row rotation matrix is the moment when the first row of photosensitive pixels in the CMOS sensor is scanned;

[0022] The positioning timestamp of the intermediate rotation matrix is the moment when the intermediate row photosensitive pixels in the CMOS sensor are scanned;

[0023] The positioning timestamp of the last row rotation matrix is the moment when the last row photosensitive pixels in the CMOS sensor are scanned;

[0024] Wherein, the intermediate row photosensitive pixels are a row of photosensitive pixels located between the first row photosensitive pixels and the last row photosensitive pixels.

[0025] Optionally, the panoramic image processing method further includes:

[0026] Using the normalization formula t = (t’ - t 1 ) / (t 3 - t 1 ), normalize the acquisition timestamps of the pixels of the panoramic image to obtain the normalized time;

[0027] The step of calculating the target correction pixel by weighted averaging the first correction pixel and the second correction pixel for each pixel of the spherical image includes:

[0028] When 0 ≤ t < 0.5, calculate the position coordinates of the target correction pixel using the first formula to obtain the target correction pixel;

[0029] When 0.5 < t ≤ 1, calculate the position coordinates of the target correction pixel using the second formula to obtain the target correction pixel;

[0030] Wherein, the first formula is p” = (1 - 2t)p 1 + 2t × p 2 ;

[0031] The second formula is p” = (1 - 2(t - 0.5))p 2 + 2(t - 0.5)p 3 ;

[0032] t represents the normalized time;

[0033] t’ represents the acquisition timestamp of the pixel of the panoramic image;

[0034] t 1 represents the positioning timestamp of the first row rotation matrix;

[0035] t 3 represents the positioning timestamp of the last row rotation matrix;

[0036] p 1Indicates the position coordinates of the pixels on the spherical image after moving according to the attitude represented by the first row rotation matrix;

[0037] p 2 Indicates the position coordinates of the pixels on the spherical image after moving according to the attitude represented by the middle rotation matrix, and the positioning timestamp of the middle rotation matrix is the moment when the photosensitive pixels in the middle row of the CMOS sensor are scanned;

[0038] p 3 Indicates the position coordinates of the pixels on the spherical image after moving according to the attitude represented by the last row rotation matrix;

[0039] p” represents the position coordinates of the target corrected pixel.

[0040] Optionally, the obtaining the acquisition timestamp of the pixels of the panoramic image includes:

[0041] When scanning a row of photosensitive pixels in the CMOS sensor, take the time of scanning this row of photosensitive pixels as the acquisition timestamp of the pixels on the fisheye image obtained by scanning this row of photosensitive pixels;

[0042] Establish a mapping relationship between the acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image to obtain a pixel-time mapping relationship, and the acquisition timestamps of the mutually matching pixels of the fisheye image and the panoramic image are the same;

[0043] Before determining the first corrected pixel and the second corrected pixel from multiple corrected pixels for each pixel of the spherical image according to the acquisition timestamp of the pixel of the spherical image, the panoramic image processing method further includes:

[0044] Determine the acquisition timestamp of the pixel of the panoramic image from the pixel-time mapping relationship according to the pixel of the panoramic image.

[0045] Optionally, the establishing a mapping relationship between the acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image to obtain a pixel-time mapping relationship includes:

[0046] Project the acquisition timestamp of the pixel of the fisheye image to the position where the pixel of the panoramic image is located to obtain a time index map, where the pixel position of the time index map records the acquisition timestamp of the pixel of the panoramic image, and the pixel position of the time index map matches the position of the pixel of the panoramic image;

[0047] The determining the acquisition timestamp of the pixel of the panoramic image from the pixel-time mapping relationship according to the pixel of the panoramic image includes:

[0048] Determine the acquisition timestamp of the pixels of the panoramic image from the time index map according to the positions of the pixels of the panoramic image.

[0049] Optionally, the obtaining a plurality of rotation matrices includes:

[0050] Obtain the accelerometer value and the angular velocity value of the panoramic camera at each of the positioning timestamps;

[0051] Use an extended Kalman filter to combine the accelerometer value and the angular velocity value to calculate the rotation matrix of the attitude of the panoramic camera relative to the initial moment at each of the positioning timestamps.

[0052] Optionally, the panoramic image is a frame image of a panoramic video stream.

[0053] To achieve this purpose, the embodiments of the present invention also adopt the following technical solutions:

[0054] A panoramic camera, comprising:

[0055] A first acquisition module, configured to acquire a panoramic image and the acquisition timestamp of the pixels of the panoramic image;

[0056] A second acquisition module, configured to acquire a plurality of rotation matrices, wherein the panoramic image is acquired by the panoramic camera in a target period, and the plurality of rotation matrices represent the attitudes of the panoramic camera relative to the initial moment at different positioning timestamps, and the positioning timestamps are within the target period;

[0057] A conversion module, configured to convert the panoramic image into a spherical image, and the acquisition timestamps of the pixels that match each other on the panoramic image and the spherical image are the same;

[0058] A correction module, configured to respectively use the plurality of rotation matrices to correct the positions of each pixel of the spherical image to obtain a plurality of corrected pixels of each pixel, and the positions of the different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the attitudes represented by the different rotation matrices;

[0059] A determination module, configured to, for each pixel of the spherical image, determine a first corrected pixel and a second corrected pixel from the plurality of corrected pixels according to the acquisition timestamp of the pixel of the spherical image, wherein the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel;

[0060] A calculation module is configured to perform weighted average calculation on each pixel of the spherical image using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel. In the weighted average calculation, the relationship between the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the relationship between the first duration and the second duration. The first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix.

[0061] Optionally, a plurality of CMOS (Complementary Metal Oxide Semiconductor) sensors are provided on the panoramic camera. One CMOS sensor is used to acquire one fisheye image, and the positioning timestamp is the moment when a row of photosensitive pixels in the CMOS sensor is scanned.

[0062] The first acquisition module includes an acquisition unit, a stitching unit, and a timestamp acquisition unit.

[0063] The acquisition unit is configured to synchronously acquire images through the plurality of CMOS sensors during the target period to obtain a plurality of the fisheye images.

[0064] The stitching unit is configured to stitch the plurality of fisheye images to obtain a panoramic image.

[0065] The timestamp acquisition unit is configured to acquire the acquisition timestamp of the pixels of the panoramic image.

[0066] The panoramic camera further includes a projection module.

[0067] The projection module is configured to project each target corrected pixel of the spherical image onto a plane to obtain a corrected panoramic image.

[0068] Optionally, the plurality of rotation matrices are respectively a first row rotation matrix, a middle row rotation matrix, and a last row rotation matrix.

[0069] The positioning timestamp of the first row rotation matrix is the moment when the first row of photosensitive pixels in the CMOS sensor is scanned.

[0070] The positioning timestamp of the middle row rotation matrix is the moment when the middle row of photosensitive pixels in the CMOS sensor is scanned.

[0071] The positioning timestamp of the last row rotation matrix is the moment when the last row of photosensitive pixels in the CMOS sensor is scanned.

[0072] Wherein, the middle row of photosensitive pixels is a row of photosensitive pixels located between the first row of photosensitive pixels and the last row of photosensitive pixels.

[0073] Optionally, the panoramic camera further includes a normalization module;

[0074] The normalization module is configured to normalize the acquisition timestamps of the pixels of the panoramic image using the normalization formula t = (t’ - t 1 ) / (t 3 -t 1 ) to obtain the normalized time;

[0075] The calculation module is further configured to calculate the position coordinates of the target correction pixel using the first formula to obtain the target correction pixel when 0 ≤ t < 0.5;

[0076] The calculation module is further configured to calculate the position coordinates of the target correction pixel using the second formula to obtain the target correction pixel when 0.5 < t ≤ 1;

[0077] wherein, the first formula is p” = (1 - 2t)p 1 + 2t×p 2 ;

[0078] The second formula is p” = (1 - 2(t - 0.5))p 2 + 2(t - 0.5)p 3 ;

[0079] t represents the normalized time;

[0080] t’ represents the acquisition timestamp of the pixel of the panoramic image;

[0081] t 1 represents the positioning timestamp of the first row rotation matrix;

[0082] t 3 represents the positioning timestamp of the last row rotation matrix;

[0083] p 1 represents the position coordinates of the pixel on the spherical image after moving according to the attitude represented by the first row rotation matrix;

[0084] p 2 represents the position coordinates of the pixel on the spherical image after moving according to the attitude represented by the middle rotation matrix, and the positioning timestamp of the middle rotation matrix is the moment when the photosensitive pixels in the middle row of the CMOS sensor are scanned;

[0085] p 3 represents the position coordinates of the pixel on the spherical image after moving according to the attitude represented by the last row rotation matrix;

[0086] p” represents the position coordinates of the target correction pixel.

[0087] Optionally, the timestamp acquisition unit includes a time determination subunit and an establishment subunit;

[0088] The time determination subunit is configured to, when scanning one row of photosensitive pixels in the CMOS sensor, use the time of scanning the row of photosensitive pixels as the acquisition timestamp of the pixels on the fisheye image obtained by scanning the row of photosensitive pixels;

[0089] The establishment subunit is configured to establish a mapping relationship between the acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image to obtain a pixel-time mapping relationship, where the acquisition timestamps of the mutually matching pixels of the fisheye image and the panoramic image are the same;

[0090] The panoramic camera further includes a timestamp determination module;

[0091] The timestamp determination module is configured to determine the acquisition timestamp of the pixels of the panoramic image from the pixel-time mapping relationship according to the pixels of the panoramic image.

[0092] Optionally, the establishment subunit is further configured to project the acquisition timestamp of the pixels of the fisheye image to the position where the pixels of the panoramic image are located to obtain a time index map, where the pixel positions of the time index map record the acquisition timestamps of the pixels of the panoramic image, and the pixel positions of the time index map match the positions of the pixels of the panoramic image;

[0093] The timestamp determination module is further configured to determine the acquisition timestamp of the pixels of the panoramic image from the time index map according to the positions of the pixels of the panoramic image.

[0094] Optionally, the second acquisition module includes a data acquisition unit and a rotation matrix calculation unit;

[0095] The data acquisition unit is configured to acquire the accelerometer value and the angular velocity value of the panoramic camera at each of the positioning timestamps;

[0096] The rotation matrix calculation unit is configured to calculate the rotation matrix of the attitude of the panoramic camera relative to the initial moment at each of the positioning timestamps by using an extended Kalman filter in combination with the accelerometer value and the angular velocity value.

[0097] Optionally, the panoramic image is a frame image of a panoramic video stream.

[0098] It can be seen from the above technical solutions that the embodiments of the present invention have the following advantages:

[0099] In the panoramic image processing method according to the embodiment of the present invention, a panoramic image and the acquisition timestamp of the pixels of the panoramic image are obtained, and a plurality of rotation matrices are obtained. The panoramic image is acquired by a panoramic camera during a target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period. The panoramic image is converted into a spherical image, and the acquisition timestamps of the mutually matching pixels on the panoramic image and the spherical image are the same. For each pixel of the spherical image, the position is corrected respectively using a plurality of rotation matrices to obtain a plurality of corrected pixels for each pixel. The positions of the different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the poses represented by different rotation matrices, so as to realize the position correction of different degrees for the pixels on the spherical image. For each pixel of the spherical image, according to the acquisition timestamp of the pixel of the spherical image, a first corrected pixel and a second corrected pixel are determined from the plurality of corrected pixels, where the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel. The first corrected pixel and the second corrected pixel are more suitable for correcting the currently corrected pixel of the spherical image. For each pixel of the spherical image, a weighted average calculation is performed using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, where in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of the first duration and the second duration. The first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix. In this way, through the weighted average calculation, the correction effects of the first corrected pixel and the second corrected pixel are combined, so that if the panoramic camera shakes when acquiring the panoramic image during the target period, through the panoramic image processing method according to the embodiment of the present invention, each pixel of the spherical image can be corrected as much as possible to the position where the pixel should be at the initial moment, thus realizing a good anti-shake effect. Description of the Drawings

[0100] Figure 1 Schematic structural diagram of a panoramic camera provided by an embodiment of the present invention;

[0101] Figure 2 Schematic flowchart of a panoramic image processing method provided by another embodiment of the present invention;

[0102] Figure 3 Schematic flowchart of a panoramic image processing method provided by another embodiment of the present invention;

[0103] Figure 4 For Figure 3 simplified flowchart of the panoramic image processing method of the illustrated embodiment;

[0104] Figure 5 Schematic diagram of the scanning process of a CMOS sensor provided by another embodiment of the present invention;

[0105] Figure 6 Mosaic diagram with rolling shutter effect provided by another embodiment of the present invention;

[0106] Figure 7 Mosaic diagram without rolling shutter effect provided by another embodiment of the present invention;

[0107] Figure 8 Schematic diagram of the structure of a panoramic camera provided by another embodiment of the present invention. Detailed implementation manners

[0108] Embodiments of the present invention provide a panoramic image processing method and a panoramic camera for improving the anti-shake effect of the panoramic camera.

[0109] Figure 1 Schematic diagram of the structure of a panoramic camera provided by an embodiment of the present invention. The panoramic camera is provided with at least two lenses. For example, Figure 1 the illustrated panoramic camera is provided with four lenses. The panoramic camera is used to collect panoramic images. Specifically, the panoramic camera can collect fisheye images through the lenses, and by stitching the fisheye images collected by each lens on the panoramic camera, a panoramic image can be obtained. Among them, a panoramic image (English name: PANORAMIC PHOTO, or PANORAMA) generally refers to an image taken above the normal effective viewing angle of a person's two eyes (about 90 degrees horizontally and 70 degrees vertically) or including the peripheral vision angle of the two eyes (about 180 degrees horizontally and 90 degrees vertically) or even a 360-degree complete scene range.

[0110] Figure 2 Flowchart of a panoramic image processing method provided by another embodiment of the present invention.

[0111] Figure 2 The panoramic image processing method of the illustrated embodiment can be applied to a panoramic camera, for example, applied to Figure 1 the illustrated panoramic camera. It should be understood that the panoramic image processing method of the embodiments of the present invention can also be applied to other processing devices, such as a server or a computer.

[0112] Referring to Figure 2 , the panoramic image processing method of the embodiments of the present invention includes the following steps:

[0113] Step 201: Obtain a panoramic image and the acquisition timestamp of the pixels of the panoramic image.

[0114] The processing device acquires a panoramic image and the acquisition timestamps of the pixels of the panoramic image. The panoramic image is composed of pixels, and each pixel has a corresponding acquisition timestamp. The acquisition timestamp of the pixel of the panoramic image is the moment when the panoramic camera acquires the pixel.

[0115] Step 202: Acquire a plurality of rotation matrices.

[0116] The processing device acquires rotation matrices, where the panoramic image is acquired by the panoramic camera during a target period, and the plurality of rotation matrices represent the postures of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period.

[0117] In the embodiment of the present invention, the acquisition of the panoramic image takes a period of time, that is, the panoramic camera acquires the panoramic image during the target period. The spatial position of the panoramic camera may change during the target period, and the spatial position of the panoramic camera can be recorded through the rotation matrices. A rotation matrix represents the posture of the panoramic camera at a certain positioning timestamp relative to the initial moment. The plurality of rotation matrices can be used to record the multiple spatial position postures of the panoramic camera during the process of acquiring the panoramic image.

[0118] Step 203: Convert the panoramic image into a spherical image.

[0119] The processing device can convert the panoramic image into a spherical image, and the acquisition timestamps of the pixels that match each other on the panoramic image and the spherical image are the same. That is, if a pixel on the spherical image is obtained by converting a pixel of the panoramic image, then the two pixels have the same acquisition timestamp. In this way, through the acquisition timestamp of the pixel of the panoramic image, the acquisition timestamp of the pixel of the spherical image can be obtained.

[0120] Step 204: For each pixel of the spherical image, use a plurality of rotation matrices to perform position correction respectively to obtain a plurality of corrected pixels for each pixel.

[0121] Among them, the positions of different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the postures represented by different rotation matrices.

[0122] Specifically, for a pixel of the spherical image, use a rotation matrix to perform position correction, that is, move the pixel according to the posture represented by the rotation matrix to obtain a corrected pixel. Among them, the position of the corrected pixel of the pixel of the spherical image is corrected. For the same pixel, perform the foregoing operation using a plurality of rotation matrices, and a plurality of corrected pixels can be obtained. For each pixel of the spherical image, use a plurality of rotation matrices to perform position correction respectively, then a plurality of corrected pixels can be obtained for each pixel of the spherical image.

[0123] Step 205: For each pixel of the spherical image, determine a first corrected pixel and a second corrected pixel from multiple corrected pixels according to the acquisition timestamp of the pixel of the spherical image.

[0124] Wherein, the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix. The first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel.

[0125] The pixels of the spherical image record acquisition timestamps, and the rotation matrices correspond to positioning timestamps. The closer the positioning timestamp is to the acquisition timestamp, the more accurate the correction of the pixel by the rotation matrix. For each pixel of the spherical image, a first corrected pixel and a second corrected pixel are respectively determined. For the currently corrected pixel, since the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, thus, the position correction of the currently corrected pixel by the first corrected pixel and the second corrected pixel is relatively accurate.

[0126] Step 206: For each pixel of the spherical image, perform a weighted average calculation using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel.

[0127] Wherein, in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of the first duration and the second duration. The first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix.

[0128] To further improve the position correction accuracy of the currently corrected pixel of the spherical image, the weights of the first corrected pixel and the second corrected pixel in the correction process can be adjusted. Specifically, the closer the acquisition timestamp of the currently corrected pixel is to the positioning timestamp of the rotation matrix, the better the effect of using the rotation matrix to perform position correction on the currently corrected pixel. Thus, among the first duration and the second duration, the smaller the duration, the closer the positioning timestamp of the rotation matrix corresponding to the duration is to the acquisition timestamp of the currently corrected pixel, and thus the greater the correction ratio of the rotation matrix corresponding to the duration to the currently corrected pixel. In other words, if the first duration is smaller than the second duration, the weight of the first corrected pixel is greater than the weight of the second corrected pixel. Similarly, if the first duration is larger than the second duration, the weight of the first corrected pixel is smaller than the weight of the second corrected pixel. In this way, after performing a weighted average calculation using the first corrected pixel and the second corrected pixel, the obtained target corrected pixel can be corrected as much as possible to the position where the pixel should be at the initial moment.

[0129] In summary, in the panoramic image processing method according to the embodiment of the present invention, a panoramic image and the acquisition timestamps of the pixels of the panoramic image are obtained, and a plurality of rotation matrices are obtained, where the panoramic image is acquired by a panoramic camera during a target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period. The panoramic image is converted into a spherical image, and the acquisition timestamps of the mutually matching pixels on the panoramic image and the spherical image are the same. For each pixel of the spherical image, the positions are corrected respectively using a plurality of rotation matrices to obtain a plurality of corrected pixels for each pixel. The positions of the different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the poses represented by different rotation matrices, so as to realize different degrees of position correction for the pixels on the spherical image. For each pixel of the spherical image, according to the acquisition timestamp of the pixel of the spherical image, a first corrected pixel and a second corrected pixel are determined from the plurality of corrected pixels, where the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel. The first corrected pixel and the second corrected pixel are more suitable for correcting the currently corrected pixel of the spherical image. For each pixel of the spherical image, a weighted average calculation is performed using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, where, in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of the first duration and the second duration. The first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix. In this way, through the weighted average calculation, the correction effects of the first corrected pixel and the second corrected pixel are combined, so that if the panoramic camera shakes when acquiring the panoramic image during the target period, through the panoramic image processing method according to the embodiment of the present invention, each pixel of the spherical image can be corrected as much as possible to the position where the pixel should be at the initial moment, thereby realizing a good anti-shake effect.

[0130] Figure 3 FIG. is a schematic flowchart of a panoramic image processing method provided by another embodiment of the present invention, where Figure 3 The panoramic image processing method of the illustrated embodiment can be based on Figure 2 The panoramic image processing method of the illustrated embodiment is implemented. Figure 3 The panoramic image processing method of the illustrated embodiment can be applied to a panoramic camera, for example, applied to Figure 1On the panoramic camera of the illustrated embodiment. It should be understood that the panoramic image processing method of the embodiments of the present invention can also be applied to other processing devices, such as servers or computers.

[0131] Taking the application to a panoramic camera as an example, the Figure 3 panoramic image processing method of the illustrated embodiment will be described in detail. To more intuitively understand the panoramic image processing method of the embodiments of the present invention, the embodiments of the present invention also provide a simplified flow of the panoramic image processing method, as Figure 4 shown. In the panoramic image processing method of the embodiments of the present invention, after obtaining multiple fisheye images, the multiple fisheye images are stitched into a panoramic image, and then the panoramic image is converted into a spherical image. The position of the pixels of the spherical image is corrected using the panoramic image processing method of the embodiments of the present invention to obtain target corrected pixels. The corrected spherical image is composed of the target corrected pixels. Then, the corrected spherical image is projected onto a plane to obtain the corrected panoramic image. Figure 3 The panoramic image processing method of the illustrated embodiment is Figure 4 a detailed implementation manner of the panoramic image processing method of the illustrated embodiment.

[0132] Referring to Figure 3 , the panoramic image processing method of the embodiments of the present invention includes the following steps:

[0133] Step 301: Synchronously collect images through multiple CMOS sensors during a target period to obtain multiple fisheye images.

[0134] In the embodiments of the present invention, a panoramic camera is provided with multiple Complementary Metal-Oxide-Semiconductor (CMOS) sensors, and one CMOS sensor is used to collect one fisheye image.

[0135] The panoramic camera synchronously collects images through multiple CMOS sensors during a target period, and one CMOS sensor collects one fisheye image. In this way, multiple CMOS sensors collect multiple fisheye images.

[0136] Step 302: Stitch the multiple fisheye images to obtain a panoramic image.

[0137] For the multiple fisheye images synchronously collected during the target period, the panoramic camera can stitch the multiple fisheye images to obtain a panoramic image.

[0138] Step 303: Obtain the acquisition timestamps of the pixels of the panoramic image.

[0139] In the embodiments of the present invention, the panoramic camera also needs to obtain the acquisition timestamps of the pixels of the panoramic image.

[0140] The acquisition timestamp of the pixels of the panoramic image is used to record the time when the panoramic camera acquires the pixels on the panoramic image.

[0141] In the embodiment of the present invention, the steps of obtaining the panoramic image and the acquisition timestamp of the pixels of the panoramic image include steps 301 to 303. Through steps 301 to 303, the panoramic camera realizes obtaining the panoramic image and the acquisition timestamp of the pixels of the panoramic image.

[0142] Step 304: Obtain a plurality of rotation matrices.

[0143] The panoramic camera obtains a plurality of rotation matrices. Among them, the panoramic image is acquired by the panoramic camera in the target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment. The positioning timestamp is within the target period, and the positioning timestamp is the moment when a row of photosensitive pixels in the scanning CMOS sensor is scanned.

[0144] In other words, when the panoramic camera synchronously acquires images through a plurality of CMOS sensors in the target period, each positioning timestamp is determined. Then, the rotation matrix at each positioning timestamp within the target period is obtained to obtain a plurality of rotation matrices. The rotation matrix can record the spatial position and pose of the panoramic camera. Specifically, at the initial moment, the panoramic camera is in a specific spatial position and pose, which can be called the first spatial position and pose. At the positioning timestamp, the panoramic camera may rotate, for example, jitter occurs during image acquisition. Therefore, at the positioning timestamp, the panoramic camera is in a specific spatial position and pose, which can be called the second spatial position and pose. The rotation matrix represents the rotation amount of the second spatial position and pose relative to the first spatial position and pose, that is, the panoramic camera in the first spatial position and pose can reach the second spatial position and pose according to the data movement of the rotation matrix.

[0145] There are various specific implementation manners of step 304. For example, an external device monitors and measures the panoramic camera to calculate the rotation matrix.

[0146] In a specific implementation manner, step 304 specifically includes: obtaining the accelerometer value and the angular velocity value of the panoramic camera at each positioning timestamp. Then, using the extended Kalman filter to combine the accelerometer value and the angular velocity value, calculate the rotation matrix of the panoramic camera at each positioning timestamp relative to the pose at the initial moment.

[0147] Since the target period is a duration, there are multiple positioning timestamps within the target period. For each positioning timestamp, the rotation matrix is calculated separately.

[0148] To better understand the panoramic image processing method according to the embodiments of the present invention, the rotation matrix will be exemplarily described below. Specifically, in a specific example, multiple rotation matrices are respectively the first-row rotation matrix, the middle-row rotation matrix, and the last-row rotation matrix. Among them, the positioning timestamp of the first-row rotation matrix is the moment when the first-row photosensitive pixels in the scanned CMOS sensor are scanned. The positioning timestamp of the middle-row rotation matrix is the moment when the middle-row photosensitive pixels in the scanned CMOS sensor are scanned. The positioning timestamp of the last-row rotation matrix is the moment when the last-row photosensitive pixels in the scanned CMOS sensor are scanned. Among them, the middle-row photosensitive pixels are a row of photosensitive pixels located between the first-row photosensitive pixels and the last-row photosensitive pixels. For example, the middle-row photosensitive pixels can be the photosensitive pixels in the middle row of the CMOS sensor.

[0149] For example, as Figure 5 shown, the photosensitive pixels of the CMOS sensor are arranged in an array form. When the CMOS sensor acquires an image, it sequentially scans the photosensitive pixels row by row. Starting from the first-row photosensitive pixels, it scans row by row, scans to the middle-row photosensitive pixels, and until it scans to the last-row photosensitive pixels. Scanning from the first-row photosensitive pixels to the last-row photosensitive pixels takes a target period. Among them, when scanning the first-row photosensitive pixels, the current time is used as a positioning timestamp, and the first-row rotation matrix is obtained. Similarly, when scanning the middle-row photosensitive pixels in the CMOS sensor, the current time is determined as a positioning timestamp, and the middle-row rotation matrix is obtained. When scanning the last-row photosensitive pixels in the CMOS sensor, the current time is determined as a positioning timestamp, and the last-row rotation matrix is obtained.

[0150] Step 305: Convert the panoramic image into a spherical image.

[0151] The panoramic camera has acquired a panoramic image, the acquisition timestamp of the pixels of the panoramic image, and multiple rotation matrices. At this time, the panoramic image may have a jello effect due to reasons such as jitter when the panoramic camera acquires the image. Therefore, it is necessary to correct the positions of the pixels of the panoramic image.

[0152] The panoramic camera can convert the panoramic image into a spherical image. The spherical image is a panoramic image presented in a spherical coordinate system. Among them, the acquisition timestamps of the mutually matching pixels on the panoramic image and the spherical image are the same, that is, if a pixel on the spherical image is obtained by converting the coordinate system of a pixel on the panoramic image, then these two pixels are mutually matching pixels, and the acquisition timestamps of these two pixels are the same.

[0153] For an example of the jello effect, reference can be made to Figure 6 and Figure 7 . Among them, Figure 6 is a stitched image with a jello effect provided by the embodiments of the present invention, Figure 7The stitched image without jello effect provided by the embodiment of the present invention. The panoramic camera uses a CMOS sensor and operates in a rolling shutter mode. When the panoramic camera captures a panoramic image, as Figure 5 shown, the photosensitive pixels of the CMOS sensor on the panoramic camera are exposed and scanned row by row in sequence. At the beginning of the exposure, the pixel points of the CMOS sensor are scanned row by row and exposed row by row until all pixel points are exposed. This process is completed in a very short time and generally does not affect the shooting. At this time, the obtained image is as Figure 7 shown, Figure 7 is the stitched image without jello effect. However, during the shooting process, if the object being photographed moves at a high speed or vibrates rapidly relative to the panoramic camera, for example, the panoramic camera shakes. At this time, when shooting with the rolling shutter mode, the row-by-row scanning speed is insufficient, and the shooting results may show situations such as "tilting", "swinging", or "partial exposure". This phenomenon is the jello effect. Using the image with the jello effect for stitching, the obtained image is as Figure 6 shown.

[0154] Step 306: For each pixel of the spherical image, use multiple rotation matrices to perform position correction respectively to obtain multiple corrected pixels for each pixel.

[0155] After obtaining the spherical image, for each pixel of the spherical image, the panoramic camera uses multiple rotation matrices to perform position correction respectively. In this way, multiple corrected pixels can be obtained for each pixel of the spherical image. Among them, one rotation matrix performs position correction on one pixel of the spherical image to obtain one corrected pixel.

[0156] The positions of different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the postures represented by different rotation matrices. Specifically, using the rotation matrix to perform position correction on the pixels of the spherical image means moving the spatial position of the pixels of the spherical image, specifically moving according to the posture represented by the rotation matrix. Since the rotation matrix represents the movement data of the panoramic camera from the initial moment to the positioning timestamp, the pixels of the spherical image are moved according to the posture represented by the rotation matrix, which can correct the spatial position of the pixels of the spherical image to be close to the position where the pixel should be at the initial moment. The position where the pixel should be at the initial moment, that is, if the orientation of the panoramic camera lens does not move, the spatial position where the pixel collected by the panoramic camera at the initial moment is located, and this spatial position can be represented in the three-dimensional coordinate system where the spherical image is located.

[0157] Similarly, at the initial moment, due to reasons such as jitter, the lens orientation of the panoramic camera moves, resulting in a position deviation when comparing the image captured by the panoramic camera with the image captured when the lens orientation of the panoramic camera does not move. The method of the embodiment of the present invention moves the pixels of the spherical image according to the attitude represented by the rotation matrix, which can synchronously move the spatial position of the pixels and the lens orientation, reduce the aforementioned position deviation, and thus slow down or even eliminate the influence brought by the jitter of the panoramic camera on the panoramic image.

[0158] In the embodiment of the present invention, optionally, the step of obtaining the acquisition timestamp of the pixels of the panoramic image specifically includes: when scanning a row of photosensitive pixels in the CMOS sensor, taking the time of scanning this row of photosensitive pixels as the acquisition timestamp of the pixels on the fisheye image obtained by scanning this row of photosensitive pixels. Then, establish a mapping relationship between the acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image to obtain a pixel-time mapping relationship, and the acquisition timestamps of the mutually matching pixels of the fisheye image and the panoramic image are the same.

[0159] Since the photosensitive pixels on the CMOS sensor are arranged in an array, in the specific photosensitive process, the photosensitive pixels of the CMOS sensor are photosensitive row by row to collect specific pixels, as Figure 5 shown. Thus, for a row of photosensitive pixels on the CMOS sensor, when scanning this row of photosensitive pixels at an acquisition timestamp, taking this acquisition timestamp as the acquisition timestamp of the pixels obtained by scanning this row of photosensitive pixels. Among them, after scanning a CMOS sensor, the obtained pixels form a fisheye image. In this way, the acquisition timestamps of the pixels on the fisheye image can be obtained.

[0160] In order to correspond the pixels of the panoramic image with the acquisition timestamps of the pixels to facilitate determining the acquisition timestamp of a specific pixel. The acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image can be established with a mapping relationship to obtain a pixel-time mapping relationship. Since the panoramic image is obtained by stitching multiple fisheye images, thus, the acquisition timestamps of the mutually matching pixels of the fisheye image and the panoramic image are the same. That is, if a pixel of the panoramic image is transformed from a pixel of the fisheye image, then these two pixels are mutually matching, and the acquisition timestamps of these two pixels are the same.

[0161] In this way, before performing step 307, the method of the embodiment of the present invention further includes: determining the acquisition timestamp of the pixels of the panoramic image from the pixel-time mapping relationship according to the pixels of the panoramic image. In this way, through the pixel-time mapping relationship, the acquisition timestamp of a specific pixel can be determined.

[0162] In the embodiment of the present invention, there are various specific implementation manners for establishing the pixel-time mapping relationship.

[0163] In a specific implementation, the steps of establishing a mapping relationship between the acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image to obtain the pixel-time mapping relationship specifically include: projecting the acquisition timestamps of the pixels of the fisheye image to the positions where the pixels of the panoramic image are located to obtain a time index map. Among them, the pixel positions of the time index map record the acquisition timestamps of the pixels of the panoramic image, and the pixel positions of the time index map match the positions of the pixels of the panoramic image.

[0164] Correspondingly, the steps of determining the acquisition timestamp of the pixel of the panoramic image from the pixel-time mapping relationship according to the pixel of the panoramic image specifically include: determining the acquisition timestamp of the pixel of the panoramic image from the time index map according to the position of the pixel of the panoramic image.

[0165] In another specific implementation, the panoramic camera can establish a specific data structure to record the pixel and the acquisition timestamp of the pixel at the same time. Specifically, when scanning a row of photosensitive pixels in the CMOS sensor, the time of scanning this row of photosensitive pixels is used as the acquisition timestamp of the pixels on the fisheye image obtained by scanning this row of photosensitive pixels. The formed data is recorded in the form of "(pixel, acquisition timestamp)", for example, the formed data is recorded in the form of "RGBT", where RGB is the pixel color and T is the acquisition timestamp of the pixel. Then, when stitching multiple fisheye images to obtain a panoramic image, the acquisition timestamp information is also recorded in the panoramic image, specifically in the form of "(pixel, acquisition timestamp)", for example, in the form of "RGBT", where RGB is the pixel color and T is the acquisition timestamp of the pixel.

[0166] When the positioning timestamp of the rotation matrix is the same as the acquisition timestamp of the pixel of the spherical image, after the pixel on the spherical image moves according to the attitude represented by the rotation matrix, it can be accurately located at the position where the pixel should be at the initial moment. If the positioning timestamp of the rotation matrix is different from the acquisition timestamp of the pixel of the spherical image, it is possible that after the pixel on the spherical image moves according to the attitude represented by the rotation matrix, it deviates from the position where the pixel should be at the initial moment. For this reason, the panoramic image processing method of the embodiments of the present invention further includes the following steps.

[0167] Step 307: For each pixel of the spherical image, determine a first correction pixel and a second correction pixel from multiple correction pixels according to the acquisition timestamp of the pixel of the spherical image.

[0168] The panoramic camera obtains the acquisition timestamp with pixels. For each pixel of the spherical image, according to the acquisition timestamp of the pixel of the spherical image, the first corrected pixel and the second corrected pixel are determined from multiple corrected pixels. Among them, the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix. The first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel.

[0169] Specifically, for a pixel of the spherical image, the pixel has an acquisition timestamp. The positioning timestamp of the rotation matrix may or may not coincide with the acquisition timestamp. The rotation matrix whose positioning timestamp is closer to the acquisition timestamp of the pixel corrects the position of the pixel more accurately. When the acquisition timestamp of the pixel is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, among multiple rotation matrices, the first rotation matrix and the second rotation matrix correct the position of the pixel more accurately than other rotation matrices. Therefore, the first corrected pixel and the second corrected pixel are relatively accurate position correction results of the pixel. In the embodiment of the present invention, the first corrected pixel and the second corrected pixel are respectively determined for each pixel of the spherical image to accurately correct the position of each pixel of the spherical image.

[0170] For example, the panoramic camera first scans the photosensitive pixels in the first row of the CMOS sensor. The scanning time is the positioning timestamp of the first row rotation matrix, and the first row rotation matrix records the attitude of the panoramic camera relative to the initial moment at this positioning timestamp. Then, the panoramic camera continues to scan the photosensitive pixels of the CMOS sensor, scans the photosensitive pixels of the target row at the target acquisition timestamp, and obtains the pixels of the target row. Among them, the pixels of the target row include the target pixel, and the acquisition timestamp of the target pixel is the target acquisition timestamp. Next, the panoramic camera scans the photosensitive pixels in the middle row, and the scanning time is the positioning timestamp of the middle rotation matrix. The middle rotation matrix records the attitude of the panoramic camera relative to the initial moment at this positioning timestamp. Then, the panoramic camera scans to the photosensitive pixels in the last row, and the scanning time is the positioning timestamp of the last row rotation matrix. The last row rotation matrix records the attitude of the panoramic camera relative to the initial moment at this positioning timestamp. In this way, the target acquisition timestamp of the target pixel is between the positioning timestamp of the first row rotation matrix and the positioning timestamp of the middle rotation matrix. Using the first row rotation matrix and the middle rotation matrix to correct the position of the target pixel is more accurate than using the last row rotation matrix. Therefore, the first corrected pixel and the second corrected pixel determined by the panoramic camera for the target pixel are the two corrected pixels obtained by using the first row rotation matrix and the middle rotation matrix to correct the position of the target pixel.

[0171] Step 308: For each pixel of the spherical image, perform a weighted average calculation using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel.

[0172] For each pixel of the spherical image, the first corrected pixel and the second corrected pixel are respectively determined. Then, a weighted average calculation is performed using the first corrected pixel and the second corrected pixel. For each pixel of the spherical image, a target corrected pixel can be respectively obtained. Among them, in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of the first duration and the second duration. The first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix.

[0173] Specifically, the first corrected pixel and the second corrected pixel are relatively accurate position correction results of the currently corrected pixel. Among the first corrected pixel and the second corrected pixel, the correction effects are not necessarily the same. As described above, the rotation matrix with a positioning timestamp closer to the acquisition timestamp of the pixel corrects the position of the pixel more accurately.

[0174] Therefore, if the positioning timestamp of the first rotation matrix is closer to the acquisition timestamp of the currently corrected pixel than the positioning timestamp of the second rotation matrix, that is, the first duration is less than the second duration, the correction effect of the first corrected pixel is better than that of the second corrected pixel. Thus, in the weighted average calculation, the weight of the first corrected pixel is greater than the weight of the second corrected pixel. On the contrary, if the positioning timestamp of the second rotation matrix is closer to the acquisition timestamp of the currently corrected pixel than the positioning timestamp of the first rotation matrix, that is, the first duration is greater than the second duration, the correction effect of the second corrected pixel is better than that of the first corrected pixel. Thus, in the weighted average calculation, the weight of the first corrected pixel is less than the weight of the second corrected pixel. In this way, after performing a weighted average calculation using the first corrected pixel and the second corrected pixel, the obtained target corrected pixel can be corrected as much as possible to the position where the pixel should be at the initial moment.

[0175] Optionally, the panoramic image processing method of the embodiment of the present invention further includes: using the normalization formula t = (t’ - t 1 ) / (t 3 - t 1 ) to normalize the acquisition timestamp of the pixel of the panoramic image to obtain a normalized time.

[0176] Correspondingly, the step of performing a weighted average calculation using the first corrected pixel and the second corrected pixel for each pixel of the spherical image to obtain a target corrected pixel specifically includes:

[0177] When \(0\leq t\lt0.5\), the position coordinates of the target corrected pixel are calculated using the first formula to obtain the target corrected pixel.

[0178] When \(0.5\lt t\leq1\), the position coordinates of the target corrected pixel are calculated using the second formula to obtain the target corrected pixel.

[0179] Among them, the first formula is \(p''=(1 - 2t)p\) 1 \(+ 2t\times p\) 2 ;

[0180] The second formula is \(p''=(1 - 2(t - 0.5))p\) 2 \(+ 2(t - 0.5)p\) 3 。

[0181] \(t\) represents the normalization time. \(t'\) represents the acquisition timestamp of the pixel of the panoramic image. \(t\) 1 represents the positioning timestamp of the first row rotation matrix. \(t\) 3 represents the positioning timestamp of the last row rotation matrix. \(p\) 1 represents the position coordinates of the pixel on the spherical image after moving according to the attitude represented by the first row rotation matrix. \(p\) 2 represents the position coordinates of the pixel on the spherical image after moving according to the attitude represented by the middle rotation matrix. The positioning timestamp of the middle rotation matrix is the moment when the photosensitive pixels in the middle row of the scanning CMOS sensor are sensed. In a CMOS sensor, the distance between the photosensitive pixels in the middle row and the photosensitive pixels in the first row is equal to the distance between the photosensitive pixels in the middle row and the photosensitive pixels in the last row. \(p\) 3 represents the position coordinates of the pixel on the spherical image after moving according to the attitude represented by the last row rotation matrix. \(p''\) represents the position coordinates of the target corrected pixel.

[0182] It should be understood that in the embodiments of the present invention, the target corrected pixel includes spatial position information and color information. The spatial position information is the above-mentioned position coordinates. The panoramic image processing method in the embodiments of the present invention can correct the spatial position of the pixel to obtain the target corrected pixel, so as to achieve anti-jelly and anti-shake. For the color information of the pixel, the panoramic image processing method in the embodiments of the present invention can remain unchanged.

[0183] Step 309: Project each target corrected pixel of the spherical image onto a plane to obtain a corrected panoramic image.

[0184] For each pixel of the spherical image, the panoramic camera performs a weighted average calculation using the first corrected pixel and the second corrected pixel to obtain the target corrected pixel. After that, the panoramic camera projects each target corrected pixel of the spherical image onto a plane to obtain a corrected panoramic image. For example, it is projected using the equidistant rectangular projection method.

[0185] In the panoramic image processing method according to the embodiments of the present invention, the position of each pixel is corrected by using a plurality of the rotation matrices, and the weight of the correction is determined according to the time information, so that the jello effect generated due to reasons such as the jitter of the panoramic camera during shooting in the corrected panoramic image is reduced, and the deformation of the image is less, thereby the stitching effect of multiple images is better. Moreover, moving the pixel according to the attitude represented by the rotation matrix to implement position correction can make the anti-shake effect of the corrected panoramic image better.

[0186] Optionally, the panoramic image is a frame image of a panoramic video stream. In this way, the panoramic image processing method according to the embodiments of the present invention can remove the jello effect in the panoramic video stream, and has a good anti-shake effect. Moreover, since each frame image is corrected based on the position of the panoramic camera at the initial moment, the entire panoramic video stream can also have a good anti-shake effect.

[0187] In summary, in the panoramic image processing method according to the embodiments of the present invention, a panoramic image and acquisition timestamps of pixels of the panoramic image are obtained, and a plurality of rotation matrices are obtained. The panoramic image is acquired by a panoramic camera during a target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period. The panoramic image is converted into a spherical image, and the acquisition timestamps of the mutually matching pixels on the panoramic image and the spherical image are the same. For each pixel of the spherical image, the plurality of rotation matrices are respectively used for position correction to obtain a plurality of corrected pixels for each pixel. The positions of the different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the poses represented by the different rotation matrices, so as to achieve different degrees of position correction for the pixels on the spherical image. For each pixel of the spherical image, according to the acquisition timestamp of the pixel of the spherical image, a first corrected pixel and a second corrected pixel are determined from the plurality of corrected pixels, where the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel. The first corrected pixel and the second corrected pixel are more suitable for correcting the currently corrected pixel of the spherical image. For each pixel of the spherical image, a weighted average calculation is performed using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, where in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of a first duration and a second duration. The first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix. In this way, through the weighted average calculation, the correction effects of the first corrected pixel and the second corrected pixel are integrated, so that if the panoramic camera shakes when acquiring the panoramic image during the target period, each pixel of the spherical image can be corrected to the position where the pixel should be at the initial moment as much as possible through the panoramic image processing method according to the embodiments of the present invention, thereby achieving a good anti-shake effect.

[0188] Figure 8 FIG. is a schematic structural diagram of a panoramic camera provided by another embodiment of the present invention. Figure 8 The panoramic camera shown can be used to execute the above Figure 2 shown embodiment of the panoramic image processing method and Figure 3 shown embodiment of the panoramic image processing method. Figure 8 The panoramic camera shown can be integrated on Figure 1 shown embodiment of the panoramic camera.

[0189] Refer to Figure 8, the panoramic camera according to an embodiment of the present invention includes:

[0190] A first acquisition module 801, configured to acquire a panoramic image and an acquisition timestamp of pixels of the panoramic image;

[0191] A second acquisition module 802, configured to acquire a plurality of rotation matrices, wherein the panoramic image is acquired by the panoramic camera during a target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period;

[0192] A conversion module 803, configured to convert the panoramic image into a spherical image, and the acquisition timestamps of the pixels that match each other on the panoramic image and the spherical image are the same;

[0193] A correction module 804, configured to perform position correction on each pixel of the spherical image respectively using a plurality of rotation matrices to obtain a plurality of corrected pixels for each pixel, and the positions of different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the poses represented by different rotation matrices;

[0194] A determination module 805, configured to determine a first corrected pixel and a second corrected pixel from a plurality of corrected pixels for each pixel of the spherical image according to the acquisition timestamp of the pixel of the spherical image, wherein the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel;

[0195] A calculation module 806, configured to perform weighted average calculation on each pixel of the spherical image using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, wherein in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of the first duration and the second duration, the first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix.

[0196] Optionally, a plurality of CMOS (Complementary Metal Oxide Semiconductor) sensors are provided on the panoramic camera, and one CMOS sensor is used to acquire one fisheye image, and the positioning timestamp is the moment when a row of photosensitive pixels in the CMOS sensor is scanned;

[0197] The first acquisition module 801 includes an acquisition unit 807, a stitching unit 808, and a timestamp acquisition unit 809;

[0198] The acquisition unit 807 is used to synchronously acquire images through multiple CMOS sensors during the target time period to obtain multiple fisheye images;

[0199] The stitching unit 808 is used to stitch multiple fisheye images to obtain a panoramic image;

[0200] The timestamp acquisition unit 809 is used to acquire the acquisition timestamps of the pixels of the panoramic image;

[0201] The panoramic camera further includes a projection module 810;

[0202] The projection module 810 is used to project each target corrected pixel of the spherical image onto a plane to obtain a corrected panoramic image.

[0203] Optionally, the multiple rotation matrices are respectively the first-row rotation matrix, the middle-row rotation matrix, and the last-row rotation matrix;

[0204] The positioning timestamp of the first-row rotation matrix is the moment when the first-row photosensitive pixels in the CMOS sensor are scanned;

[0205] The positioning timestamp of the middle-row rotation matrix is the moment when the middle-row photosensitive pixels in the CMOS sensor are scanned;

[0206] The positioning timestamp of the last-row rotation matrix is the moment when the last-row photosensitive pixels in the CMOS sensor are scanned;

[0207] Wherein, the middle-row photosensitive pixels are a row of photosensitive pixels located between the first-row photosensitive pixels and the last-row photosensitive pixels.

[0208] Optionally, the panoramic camera further includes a normalization module 811;

[0209] The normalization module 811 is used to normalize the acquisition timestamps of the pixels of the panoramic image using the normalization formula t = (t’ - t 1 ) / (t 3 - t 1 ) to obtain a normalized time;

[0210] The calculation module 806 is further used to calculate the position coordinates of the target corrected pixel using the first formula to obtain the target corrected pixel when 0 ≤ t < 0.5;

[0211] The calculation module 806 is further used to calculate the position coordinates of the target corrected pixel using the second formula to obtain the target corrected pixel when 0.5 < t ≤ 1;

[0212] Wherein, the first formula is p” = (1 - 2t)p 1 + 2t × p 2 ;

[0213] The second formula is p” = (1 - 2(t - 0.5))p 2 + 2(t - 0.5)p 3 ;

[0214] t represents the normalization time;

[0215] t’ represents the acquisition timestamp of the pixels of the panoramic image;

[0216] t 1 represents the positioning timestamp of the first-row rotation matrix;

[0217] t 3 represents the positioning timestamp of the last-row rotation matrix;

[0218] p 1 represents the position coordinates of the pixels on the spherical image after moving according to the attitude represented by the first-row rotation matrix;

[0219] p 2 represents the position coordinates of the pixels on the spherical image after moving according to the attitude represented by the middle rotation matrix, and the positioning timestamp of the middle rotation matrix is the moment when the photosensitive pixels in the middle row of the scanning CMOS sensor are scanned;

[0220] p 3 represents the position coordinates of the pixels on the spherical image after moving according to the attitude represented by the last-row rotation matrix;

[0221] p” represents the position coordinates of the target corrected pixels.

[0222] Optionally, the timestamp acquisition unit 809 includes a time determination subunit 812 and an establishment subunit 813;

[0223] The time determination subunit 812 is configured to, when scanning a row of photosensitive pixels in the CMOS sensor, use the time of scanning this row of photosensitive pixels as the acquisition timestamp of the pixels on the fisheye image obtained by scanning this row of photosensitive pixels;

[0224] The establishment subunit 813 is configured to establish a mapping relationship between the acquisition timestamp of the pixels of the fisheye image and the pixels of the panoramic image to obtain a pixel time mapping relationship, and the acquisition timestamps of the mutually matching pixels of the fisheye image and the panoramic image are the same;

[0225] The panoramic camera further includes a timestamp determination module 814;

[0226] The timestamp determination module 814 is configured to determine the acquisition timestamp of the pixels of the panoramic image from the pixel time mapping relationship according to the pixels of the panoramic image.

[0227] Optionally, a sub-unit 813 is established and is further configured to project the acquisition timestamp of the pixels of the fish-eye image to the position where the pixels of the panoramic image are located, so as to obtain a time index map, where the pixel position of the time index map records the acquisition timestamp of the pixels of the panoramic image, and the pixel position of the time index map matches the position of the pixels of the panoramic image;

[0228] A timestamp determination module 814 is further configured to determine the acquisition timestamp of the pixels of the panoramic image from the time index map according to the position of the pixels of the panoramic image.

[0229] Optionally, the second acquisition module 802 includes a data acquisition unit 815 and a rotation matrix calculation unit 816;

[0230] The data acquisition unit 815 is configured to acquire the accelerometer value and the angular velocity value of the panoramic camera at each positioning timestamp;

[0231] The rotation matrix calculation unit 816 is configured to calculate the rotation matrix of the attitude of the panoramic camera relative to the initial moment at each positioning timestamp by using an extended Kalman filter in combination with the accelerometer value and the angular velocity value.

[0232] Optionally, the panoramic image is a frame image of a panoramic video stream.

[0233] In summary, in the panoramic camera according to the embodiment of the present invention, the first acquisition module 801 acquires a panoramic image and the acquisition timestamp of the pixels of the panoramic image. The second acquisition module 802 acquires a plurality of rotation matrices. Among them, the panoramic image is acquired by the panoramic camera during a target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period. The conversion module 803 converts the panoramic image into a spherical image, and the acquisition timestamps of the pixels that match each other on the panoramic image and the spherical image are the same. For each pixel of the spherical image, the correction module 804 respectively uses the plurality of rotation matrices for position correction to obtain a plurality of corrected pixels for each pixel. The positions of the different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the poses represented by different rotation matrices. For each pixel of the spherical image, the determination module 805 determines a first corrected pixel and a second corrected pixel from the plurality of corrected pixels according to the acquisition timestamp of the pixel of the spherical image, where the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel. For each pixel of the spherical image, the calculation module 806 uses the first corrected pixel and the second corrected pixel for weighted average calculation to obtain a target corrected pixel, where in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of the first duration and the second duration. The first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix. In this way, through the weighted average calculation, the correction effects of the first corrected pixel and the second corrected pixel are combined, so that if the panoramic camera shakes when acquiring the panoramic image during the target period, each pixel of the spherical image can be corrected as much as possible to the position where the pixel should be at the initial moment through the panoramic image processing method according to the embodiment of the present invention, thereby achieving a good anti-shake effect.

[0234] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0235] In several embodiments provided by this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0236] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0238] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0239] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. A panoramic image processing method, characterized in that, it includes: Obtain a panoramic image and the acquisition timestamp of the pixels of the panoramic image; Obtain a plurality of rotation matrices, wherein the panoramic image is acquired by a panoramic camera during a target period, and the plurality of rotation matrices represent the poses of the panoramic camera at different positioning timestamps relative to the initial moment, and the positioning timestamps are within the target period; Convert the panoramic image into a spherical image, and the acquisition timestamps of the mutually matching pixels on the panoramic image and the spherical image are the same; For each pixel of the spherical image, respectively use the plurality of rotation matrices for position correction to obtain a plurality of corrected pixels for each pixel, and the positions of the different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the poses represented by the different rotation matrices; For each pixel of the spherical image, determine a first corrected pixel and a second corrected pixel from the plurality of corrected pixels according to the acquisition timestamp of the pixel of the spherical image, wherein the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel; For each pixel of the spherical image, perform weighted average calculation using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, wherein in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of the first duration and the second duration, the first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix.

2. The panoramic image processing method according to claim 1, characterized in that, A plurality of CMOS (Complementary Metal Oxide Semiconductor) sensors are provided on the panoramic camera, and one CMOS sensor is used to acquire one fisheye image, and the positioning timestamp is the moment when a row of photosensitive pixels in the CMOS sensor is scanned; The obtaining of the panoramic image and the acquisition timestamp of the pixels of the panoramic image includes: Synchronously acquire images through the plurality of CMOS sensors during the target period to obtain a plurality of fisheye images; Stitch the plurality of fisheye images to obtain a panoramic image; Obtain the acquisition timestamp of the pixels of the panoramic image; After performing weighted average calculation using the first corrected pixel and the second corrected pixel for each pixel of the spherical image to obtain a target corrected pixel, the panoramic image processing method further includes: Project each target corrected pixel of the spherical image onto a plane to obtain a corrected panoramic image.

3. The panoramic image processing method according to claim 2, characterized in that, The multiple rotation matrices are respectively a first-row rotation matrix, a middle-row rotation matrix, and a last-row rotation matrix; The positioning timestamp of the first-row rotation matrix is the moment when the first-row photosensitive pixels in the CMOS sensor are scanned; The positioning timestamp of the middle-row rotation matrix is the moment when the middle-row photosensitive pixels in the CMOS sensor are scanned; The positioning timestamp of the last-row rotation matrix is the moment when the last-row photosensitive pixels in the CMOS sensor are scanned; Wherein, the middle-row photosensitive pixels are a row of photosensitive pixels located between the first-row photosensitive pixels and the last-row photosensitive pixels.

4. The panoramic image processing method according to claim 3, characterized in that the panoramic image processing method further includes: Using the normalization formula t = (t’ - t 1 ) / (t 3 - t 1 ), normalize the acquisition timestamps of the pixels of the panoramic image to obtain the normalized time; For each pixel of the spherical image, performing weighted average calculation using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, including: When 0 ≤ t < 0.5, calculating the position coordinates of the target corrected pixel using a first formula to obtain the target corrected pixel; When 0.5 < t ≤ 1, calculating the position coordinates of the target corrected pixel using a second formula to obtain the target corrected pixel; Among them, the first formula is p” = (1 - 2t)p 1 + 2t × p 2 ; The second formula is p” = (1 - 2(t - 0.5))p 2 + 2(t - 0.5)p 3 ; t represents the normalized time; t' represents the acquisition timestamp of the pixel of the panoramic image; t 1 Indicates the positioning timestamp of the first row rotation matrix; t 3 Indicates the positioning timestamp of the end row rotation matrix; p 1 representing the position coordinates after the pixels on the spherical image are moved according to the attitude represented by the first row rotation matrix; p 2 represents the position coordinates of the pixels on the spherical image after moving according to the attitude represented by the intermediate rotation matrix, and the positioning timestamp of the intermediate rotation matrix is the moment when the photosensitive pixels in the middle row of the CMOS sensor are scanned; p 3 representing the position coordinates after the pixels on the spherical image are moved according to the attitude represented by the last row rotation matrix; p'' represents the position coordinates of the target corrected pixel.

5. The panoramic image processing method according to claim 2, characterized in that the obtaining the acquisition timestamp of the pixel of the panoramic image includes: When scanning a row of photosensitive pixels in the CMOS sensor, taking the time of scanning this row of photosensitive pixels as the acquisition timestamp of the pixel on the fisheye image obtained by scanning this row of photosensitive pixels; Establishing a mapping relationship between the acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image to obtain a pixel-time mapping relationship, and the acquisition timestamps of the mutually matching pixels of the fisheye image and the panoramic image are the same; Before determining the first corrected pixel and the second corrected pixel from multiple corrected pixels for each pixel of the spherical image according to the acquisition timestamp of the pixel of the spherical image, the panoramic image processing method further includes: Determining the acquisition timestamp of the pixel of the panoramic image from the pixel-time mapping relationship according to the pixel of the panoramic image.

6. The panoramic image processing method according to claim 5, characterized in that the establishing a mapping relationship between the acquisition timestamps of the pixels of the fisheye image and the pixels of the panoramic image to obtain a pixel-time mapping relationship includes: Projecting the acquisition timestamp of the pixel of the fisheye image to the position where the pixel of the panoramic image is located to obtain a time index map, wherein the pixel position of the time index map records the acquisition timestamp of the pixel of the panoramic image, and the pixel position of the time index map matches the position of the pixel of the panoramic image; The determining the acquisition timestamp of the pixel of the panoramic image from the pixel-time mapping relationship according to the pixel of the panoramic image includes: Determining the acquisition timestamp of the pixel of the panoramic image from the time index map according to the position of the pixel of the panoramic image.

7. The panoramic image processing method according to claim 1, wherein, the obtaining of a plurality of rotation matrices includes: obtaining the accelerometer value and the angular velocity value of the panoramic camera at each of the positioning timestamps; using an extended Kalman filter to combine the accelerometer value and the angular velocity value to calculate the rotation matrix of the attitude of the panoramic camera relative to the initial moment at each of the positioning timestamps.

8. The panoramic image processing method according to claim 2, wherein, the panoramic image is a frame image of a panoramic video stream.

9. A panoramic camera, wherein, it includes: a first obtaining module, configured to obtain a panoramic image and the acquisition timestamp of the pixels of the panoramic image; a second obtaining module, configured to obtain a plurality of rotation matrices, wherein the panoramic image is acquired by the panoramic camera in a target period, and the plurality of rotation matrices represent the attitudes of the panoramic camera relative to the initial moment at different positioning timestamps, and the positioning timestamps are within the target period; a conversion module, configured to convert the panoramic image into a spherical image, and the acquisition timestamps of the mutually matching pixels on the panoramic image and the spherical image are the same; a correction module, configured to perform position correction on each pixel of the spherical image respectively using the plurality of rotation matrices to obtain a plurality of corrected pixels for each pixel, and the positions of the different corrected pixels belonging to the same pixel are the positions after the pixels on the spherical image are moved according to the attitudes represented by the different rotation matrices; a determination module, configured to determine a first corrected pixel and a second corrected pixel from the plurality of corrected pixels for each pixel of the spherical image according to the acquisition timestamp of the pixel of the spherical image, wherein the acquisition timestamp of the currently corrected pixel of the spherical image is between the positioning timestamp of the first rotation matrix and the positioning timestamp of the second rotation matrix, the first rotation matrix is the rotation matrix used to obtain the first corrected pixel, and the second rotation matrix is the rotation matrix used to obtain the second corrected pixel; a calculation module, configured to perform weighted average calculation on each pixel of the spherical image using the first corrected pixel and the second corrected pixel to obtain a target corrected pixel, wherein in the weighted average calculation, the magnitude relationship of the weights of the first corrected pixel and the second corrected pixel is inversely proportional to the magnitude relationship of a first duration and a second duration, the first duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the first rotation matrix, and the second duration is the time difference between the acquisition timestamp of the currently corrected pixel of the spherical image and the positioning timestamp of the second rotation matrix.

10. The panoramic camera according to claim 9, wherein, a plurality of CMOS (Complementary Metal Oxide Semiconductor) sensors are provided on the panoramic camera, one CMOS sensor is used to acquire one fisheye image, and the positioning timestamp is the moment when a row of photosensitive pixels in the CMOS sensor is scanned; the first obtaining module includes an acquisition unit, a splicing unit, and a timestamp obtaining unit; The acquisition unit is configured to synchronously acquire images through the multiple CMOS sensors during the target period to obtain multiple fisheye images; The stitching unit is configured to stitch the multiple fisheye images to obtain a panoramic image; The timestamp acquisition unit is configured to acquire the acquisition timestamps of the pixels of the panoramic image; The panoramic camera further includes a projection module; The projection module is configured to project each target corrected pixel of the spherical image onto a plane to obtain a corrected panoramic image.

Images