Real-time viewfinder method, panoramic camera, and computer-readable storage medium

Abstract

The application discloses a real-time viewfinder method of a panoramic camera, which comprises the following steps: generating a longitude-latitude graph mapping table according to the calibration parameters of each lens of the panoramic camera; calculating a first three-dimensional coordinate of an image displayed on a screen of a touch display screen of the panoramic camera; calculating a second three-dimensional coordinate according to the touch direction and angle of the touch display screen and the first three-dimensional coordinate; mapping the second three-dimensional coordinate into longitude and latitude; generating a real-time preview picture on the touch display screen according to the longitude and latitude and the longitude-latitude graph mapping table; wherein the longitude-latitude graph mapping table comprises the horizontal coordinate, the vertical coordinate and the weight value of each lens corresponding to any point on the sphere with the panoramic camera as the center. Compared with the prior art, the application does not need to globally splice the images of multiple lenses, but only needs to splice the images which need to be displayed on the display screen, and has the advantages of good real-time performance, convenient operation and low implementation cost.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a real-time framing method for a panoramic camera, a panoramic camera, and a computer-readable storage medium. Background Technology

[0002] Existing panoramic cameras can only capture the view directly in front of the lens. To achieve a 360-degree view, the video preview needs to be streamed to a mobile client, which then stitches the images together to create a panoramic image. Finally, the view can be captured from any angle on the mobile client's screen.

[0003] However, the above framing method has the following two main drawbacks.

[0004] 1. Real-time preview is not available.

[0005] Since transmitting the video preview stream to the client takes time and stitching the images takes a lot of time, latency is inevitably increased.

[0006] 2. It is inconvenient to operate and has a high implementation cost.

[0007] In addition, since this function requires an additional client, it is inevitably inconvenient to operate, and the real-time cost is relatively high.

[0008] Therefore, it is necessary to improve the existing real-time framing methods of panoramic cameras. Summary of the Invention

[0009] The purpose of this invention is to provide a real-time framing method for a panoramic camera, a panoramic camera, and a computer-readable storage medium, in order to overcome the shortcomings of existing technologies.

[0010] In a first aspect, the present invention provides a real-time framing method for a panoramic camera, the method comprising: A latitude and longitude mapping table is generated based on the calibration parameters of each lens of the panoramic camera; the first three-dimensional coordinates of the image displayed on the touch screen of the panoramic camera are calculated; the second three-dimensional coordinates are calculated based on the detected touch direction and angle of the touch screen and the first three-dimensional coordinates; the second three-dimensional coordinates are mapped into latitude and longitude; a real-time preview screen is generated on the touch screen based on the latitude and longitude, the latitude and longitude mapping table, and the real-time image data of each lens; wherein, the latitude and longitude mapping table includes the horizontal coordinate, vertical coordinate, and weight value of each lens corresponding to any point on the sphere centered on the panoramic camera.

[0011] Secondly, the present invention provides a panoramic camera, including a camera body, at least two lenses, and a touch screen mounted on the camera body, as well as a latitude and longitude mapping module for generating a latitude and longitude mapping table based on the calibration parameters of each lens of the panoramic camera; a first calculation module for calculating the first three-dimensional coordinates of the image displayed on the touch screen of the panoramic camera; a second calculation module for calculating the second three-dimensional coordinates based on the detected touch direction and angle of the touch screen and the first three-dimensional coordinates; a mapping module for mapping the second three-dimensional coordinates into latitude and longitude; and a preview screen module for generating a real-time preview screen on the touch screen based on the latitude and longitude, the latitude and longitude mapping table, and the acquired real-time image data of each lens; wherein the latitude and longitude mapping table includes the abscissa, ordinate, and weight value of each lens corresponding to any point on a sphere centered on the panoramic camera.

[0012] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described real-time framing method for a panoramic camera.

[0013] Compared with existing technologies, this invention does not require global stitching of images from multiple lenses. It only requires stitching the images to be displayed on the screen, enabling real-time framing from any angle on the touch screen of the panoramic camera. The direction of the screen preview can be adjusted by sliding the finger, thus better assisting the user in shooting or recording, improving the user experience, and has the advantages of good real-time performance, convenient operation and low implementation cost. Attached Figure Description

[0014] Figure 1 This is a flowchart of the real-time framing method of the panoramic camera in Embodiment 1 of the present invention.

[0015] Figure 2 This is a block diagram of the panoramic camera in Embodiment 2 of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0017] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0018] Example 1 like Figure 1 As shown, a preferred embodiment of the real-time framing method for the panoramic camera in this embodiment includes the following steps.

[0019] S1. Generate a latitude and longitude mapping table based on the calibration parameters of each lens of the panoramic camera.

[0020] The panoramic camera in this embodiment includes at least two lenses (such as fisheye lenses or wide-angle lenses), with partial overlap of the field of view between two adjacent lenses to form a 360° panoramic field of view. The process of generating the latitude and longitude mapping table in this embodiment is as follows: First, the real-time image data (video) captured by the panoramic camera is calibrated to obtain calibration parameters. The calibration parameters include the horizontal angle, pitch angle, and spin angle of each lens. Then, a corresponding lens latitude and longitude table is established for each lens, and combined with the calibration parameters to generate the latitude and longitude mapping table, so that each point in space can find its coordinates and weight corresponding to the real-time image data captured by at least one lens. The latitude and longitude mapping table includes an abscissa, a ordinate, and a weight value. The abscissa represents the longitude of the sphere centered on the panoramic camera, and the ordinate represents the latitude of the sphere centered on the panoramic camera. Here, "centered on the panoramic camera" includes the geometric center of the overall shape of the panoramic camera or the center of gravity of the panoramic camera. In this embodiment, the geometric center of the focal point of each lens of the panoramic camera can also be used as the center. For example, when there are two lenses, the center is the midpoint of the line connecting the two lens focal points; when there are three lenses, the center is the center of gravity of the triangle formed by the three lens focal points; when there are four lenses, the center is the intersection of the diagonals of the quadrilateral formed by the four lens focal points.

[0021] In the process of synthesizing a latitude and longitude map from lens latitude and longitude tables, for points simultaneously located in images generated by two adjacent lenses, weights are assigned based on their distances from the centers of the two generated images. The closer a point is to the center of an image generated by a particular lens, the larger its weight value, ensuring that the sum of the two weight values ​​equals 1. This effectively eliminates the stitching gap between the two lens images. For example, the weight value of each lens for a point can be calculated using a weighted average. Specifically, assuming a point's distance from the first lens is d1 and its distance from the second lens is d2, then the weight of the first lens is d2 / (d1+d2), and the weight of the second lens is d1 / (d1+d2). For points located in images generated by only one lens, the weight of that lens is set to 1 to ensure the sharpness of the entire spatial image. In the above process, for points simultaneously located in images generated by three or more lenses, the two lenses with the smallest distances from the point to the center of the image are selected, and different weights are assigned based on the distances of the point to the centers of the images generated by these two lenses.

[0022] S2. Calculate the first three-dimensional coordinates of the image displayed on the touch screen of the panoramic camera.

[0023] In linear projection mode, assuming the view is taken directly in front of the panoramic camera, the coordinates of the image displayed on the touch screen are calculated as follows.

[0024] For any point on the screen with screen coordinates (i, j), the first three-dimensional coordinates (X1, Y1, Z1) of the displayed image are calculated as follows: X1 = tan(PI * 0.5 - fov * PI / 180 * 0.5) * rayW * 0.5; Y1 = (j - rayW * 0.5); Z1 = (i - rayH * 0.5); where rayW represents the width of the screen, rayH represents the height of the screen, fov represents the field of view of the image displayed on the screen, and PI represents π.

[0025] Using the above method, the first three-dimensional coordinates of all points on the touch screen of the panoramic camera can be obtained.

[0026] S3. Calculate the second three-dimensional coordinates based on the detected touch direction and angle of the touch screen and the first three-dimensional coordinates.

[0027] In this step, the sliding state of a finger or stylus on the touchscreen is detected. In this embodiment, a left-right slide on the screen represents a yaw axis oscillation of the viewing angle, and a up-down slide represents a pitch axis oscillation of the viewing angle. Since the sliding of a finger or stylus on the touchscreen does not involve screen flipping, there is no roll axis oscillation, i.e., the roll axis angle remains 0. Let θ represent the detected left-right slide in arc, and φ represent the detected up-down slide in arc. Then, the rot of the 3*3 rotation matrix can be calculated. rot(0, 0) = cos(φ) * cos(θ); rot(0, 1) = -sinf(θ); rot(0, 2) = sinf(φ) * cosf(θ); rot(1, 0) = cosf(φ) * sinf(θ); rot(1, 1) = cosf(θ); rot(1, 2) = sinf(φ) * sinf(θ); rot(2, 0) = -sinf(φ); rot(2, 1) = 0; rot(2, 2) = cosf(φ); For any first three-dimensional coordinate (X1, Y1, Z1), the second three-dimensional coordinate (X2, Y2, Z2) of the displayed image is calculated as follows: X2 =X1* rot(0, 0) + Y1 * rot(0, 1) +Z1 * rot(0, 2); Y2= X1 * rot(1, 0) + Y1 * rot(1, 1) + Z1 * rot(1, 2); Z2 =X1 * rot(2, 0) + Y1 * rot(2, 1) + Z1 * rot(2, 2); Substituting the rot of the 3x3 rotation matrix, we obtain the formula for calculating the second three-dimensional coordinates (X2, Y2, Z2): X2 =X1 * cos(φ) * cos(θ) + Y1* (- sinf(θ)) + Z1* sinf(φ) * cosf(θ); Y2 =X1 *cosf(φ) * sinf(θ) + Y1 * cosf(θ) + Z1 * sinf(φ) * sinf(θ); Z2 =X1 * (-sinf(φ)) + Y1 *0 +Z1 * cosf(φ).

[0028] S4. Map the second three-dimensional coordinates to latitude and longitude.

[0029] For any second three-dimensional coordinate (X2, Y2, Z2), its latitude and longitude (fi, theta) are calculated as follows: fi = atan2f(Y2, X2); theta=PI * 0.5-atan2f(Z2,sqrt(X2*X2+Y2*Y2)); When the calculated value of fi is less than 0, the value of fi is adjusted to 2*PI-fi to ensure that fi is within the range of 0 to 2PI. Here, PI represents pi (π), and atan2f(a, b) is the arctangent of b / a in radians.

[0030] S5. Generate a real-time preview screen on the touch screen based on latitude and longitude, latitude and longitude mapping table, and real-time image data of each lens.

[0031] Based on the latitude and longitude of the second three-dimensional coordinates obtained in step S4, find the coordinates and weight values ​​of the real-time image data (video) in the latitude and longitude mapping table; then copy the corresponding color value or the weighted color value in the coordinates to the target image to obtain a real-time preview image, and then display it on the touch screen.

[0032] As can be seen from this embodiment, the present invention only needs to stitch together the content to be displayed on the touch screen of the panoramic camera, and does not need to stitch together the images of all lenses as a whole. Therefore, the processing speed is fast and it is suitable for real-time framing of panoramic cameras.

[0033] Example 2 like Figure 2 The diagram shows the structural modules of the panoramic camera in this embodiment. The panoramic camera includes two fisheye lenses (or other numbers) and a touchscreen display. The two fisheye lenses are mounted on opposite sides of the panoramic camera, with partial overlap in their fields of view to form a 360° panoramic field of view. The touchscreen display is roughly rectangular and is used to display a preview image of the panoramic camera. The panoramic camera also includes: a latitude-longitude mapping module for generating a latitude-longitude mapping table based on the calibration parameters of each lens of the panoramic camera; a first calculation module for calculating the first three-dimensional coordinates of the image displayed on the touchscreen display; a second calculation module for calculating the second three-dimensional coordinates based on the detected touch direction and angle of the touchscreen display and the first three-dimensional coordinates; a mapping module for mapping the second three-dimensional coordinates into latitude and longitude; and a preview screen module for generating a real-time preview screen on the touchscreen display based on the latitude and longitude, the latitude-longitude mapping table, and the acquired real-time image data of each lens. The latitude-longitude mapping table includes the horizontal and vertical coordinates and weight values ​​of each lens corresponding to any point on a sphere centered on the panoramic camera.

[0034] Example 3 This embodiment discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the real-time framing method of the panoramic camera in Embodiment 1.

[0035] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The storage medium can be a computer-readable storage medium, such as ferromagnetic random access memory (FRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disk-read-only memory (CD-ROM); or it can be a device that includes one or any combination of the above-mentioned memories.

[0036] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time framing method for a panoramic camera, characterized in that, include: A latitude and longitude mapping table is generated based on the calibration parameters of each lens of the panoramic camera; Calculate the first three-dimensional coordinates of the image displayed on the touch screen of the panoramic camera; The second three-dimensional coordinates are calculated based on the detected touch direction and angle of the touch display screen and the first three-dimensional coordinates; Map the second three-dimensional coordinates to latitude and longitude; A real-time preview screen is generated on the touch screen based on latitude and longitude, latitude and longitude mapping table, and real-time image data of each lens. The latitude and longitude mapping table includes the horizontal and vertical coordinates and weight values ​​of each lens corresponding to any point on the sphere centered on the panoramic camera.

2. The real-time framing method for a panoramic camera according to claim 1, characterized in that, Generating a latitude and longitude mapping table based on the calibration parameters of each lens of the panoramic camera includes: calibrating the parameters of the real-time image data captured by the panoramic camera to obtain the calibration parameters, establishing a corresponding lens latitude and longitude table for each lens, and combining the lens calibration parameters and the lens latitude and longitude table to generate a latitude and longitude mapping table.

3. The real-time framing method for a panoramic camera according to claim 1, characterized in that, The specific method of combining lens calibration parameters and lens latitude and longitude tables to generate a latitude and longitude mapping table is as follows: for points that are simultaneously located in images generated by two adjacent lenses, weights are set according to their distances from the center positions of the two images.

4. The real-time framing method for a panoramic camera according to claim 1, characterized in that, The first three-dimensional coordinates of the image displayed on the touch screen of the panoramic camera are calculated as follows: For any point on the screen with screen coordinates (i, j), the first three-dimensional coordinates (X1, Y1, Z1) of the displayed image are calculated as follows: X1=tan(PI * 0.5 - fov * PI / 180 * 0.5) * rayW*0.5; Y1=(j- rayW*0.5); Z1 = (i - rayH * 0.5); Where rayW represents the width of the screen, rayH represents the height of the screen, fov represents the field of view of the image displayed on the screen, and PI represents pi (π).

5. The real-time framing method for a panoramic camera according to claim 2, characterized in that, The method for calculating the second three-dimensional coordinates based on the detected touch direction and angle of the touch display screen and the first three-dimensional coordinates is as follows: For any first three-dimensional coordinate (X1, Y1, Z1), the method for calculating the second three-dimensional coordinates (X2, Y2, Z2) of the displayed image is as follows: X2 =X1 * cos(φ) * cos(θ) + Y1* (- sinf(θ)) + Z1* sinf(φ) * cosf(θ); Y2 =X1 *cosf(φ) * sinf(θ) + Y1 * cosf(θ) + Z1 * sinf(φ) * sinf(θ); Z2 =X1 * (-sinf(φ)) + Y1 *0 +Z1 * cosf(φ); Where θ is the detected arc of the screen sliding left and right, and φ is the detected arc of the screen sliding up and down.

6. The real-time framing method for a panoramic camera according to claim 1, characterized in that, The mapping of the second three-dimensional coordinates to latitude and longitude specifically involves the following: For any second three-dimensional coordinate (X2, Y2, Z2), its latitude and longitude (fi, theta) are calculated as follows: fi = atan2f(Y2, X2); theta=PI * 0.5-atan2f(Z2,sqrt(X2*X2+Y2*Y2)); Where PI represents π, and atan2f(a, b) is the arctangent of b / a in radians.

7. The real-time framing method for a panoramic camera according to claim 1, characterized in that, The process of generating a real-time preview screen on the touch display based on latitude and longitude, a latitude and longitude mapping table, and the acquired real-time image data from each lens includes: Based on the obtained second three-dimensional coordinates, find the coordinates of the real-time image data corresponding to each shot in the latitude and longitude mapping table; Copy the corresponding color values ​​or weighted color values ​​from each coordinate to the target image to obtain a real-time preview.

8. A panoramic camera, comprising at least two lenses and a touch display screen, characterized in that, Also includes: The latitude and longitude mapping table module is used to generate a latitude and longitude mapping table based on the calibration parameters of each lens of the panoramic camera; The first calculation module is used to calculate the first three-dimensional coordinates of the image displayed on the touch screen of the panoramic camera. The second calculation module is used to calculate the second three-dimensional coordinates based on the detected touch direction and angle of the touch display screen and the first three-dimensional coordinates; The mapping module is used to map the second three-dimensional coordinates into latitude and longitude. The preview screen module is used to generate a real-time preview screen on the touch screen based on latitude and longitude, latitude and longitude map mapping table and real-time image data of each lens. The latitude and longitude mapping table includes horizontal coordinates, vertical coordinates, and weight values.

9. The panoramic camera according to claim 8, characterized in that, Also includes: The weighted calculation module is used to calculate the coordinate value by performing a weighted average calculation based on the specific coordinates of each real-time image data and the distance from the specific coordinate to the center of the respective real-time image when the horizontal and vertical coordinates correspond to the specific coordinates of the real-time image data of multiple lenses.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the real-time framing method of the panoramic camera according to any one of claims 1 to 7.

Images