Method, device, equipment and storage medium for panoramic video recording

Abstract

Embodiments of the present application provide a panoramic video recording method, device, equipment and storage medium. The method comprises: determining an anti-distortion camera image and a spherical projection template at a current time; for each spherical coordinate point in the spherical projection template, determining a mapping point texture of the spherical coordinate point in the anti-distortion camera image according to a camera field of view angle; and performing pixel rendering on the spherical projection template according to the mapping point texture of each spherical coordinate point to obtain a panoramic video frame recorded at the current time. Embodiments of the present application can eliminate camera distortion and spherical projection distortion in the panoramic video frame, realize non-distortion recording of the panoramic video, and improve the image quality of the panoramic video recording.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for panoramic video recording. Background Technology

[0002] Currently, extended reality (XR) technology is being used in a wider range of applications, including virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0003] Typically, after wearing an XR device, users can enable the Video See-Through (VST) function to record panoramic videos of real-world scenes, allowing third-party users to view these videos using another XR device.

[0004] To ensure an immersive viewing experience for third-party users, panoramic videos are typically recorded using spherical projection, transforming each frame onto a two-dimensional plane for storage and enabling spherical playback. However, this projection process causes stretching and distortion of the panoramic video frames, and lens manufacturing and assembly variations also introduce some distortion. Therefore, there is an urgent need to design a distortion-free panoramic video recording solution to eliminate these distortions. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for panoramic video recording, enabling distortion-free recording of panoramic video. While ensuring the image quality of panoramic video recording, it reduces frame processing overhead and increases the recording frame rate of panoramic video.

[0006] In a first aspect, embodiments of this application provide a method for panoramic video recording, the method comprising:

[0007] Determine the anti-distortion camera image and spherical projection template at the current moment;

[0008] For each spherical coordinate point within the spherical projection template, the mapping point texture of the spherical coordinate point in the anti-distortion camera image is determined according to the camera's field of view.

[0009] Based on the texture of the mapped points of each spherical coordinate point, the spherical projection template is pixel-rendered to obtain the panoramic video frame recorded at the current moment.

[0010] Secondly, embodiments of this application provide a panoramic video recording apparatus, the apparatus comprising:

[0011] The image determination module is used to determine the anti-distortion camera image and the spherical projection template at the current moment;

[0012] The texture mapping module is used to determine the mapping point texture of each spherical coordinate point in the anti-distortion camera image based on the camera's field of view, for each spherical coordinate point within the spherical projection template.

[0013] The video frame recording module is used to perform pixel rendering on the spherical projection template according to the mapping point texture of each spherical coordinate point to obtain the panoramic video frame recorded at the current moment.

[0014] Thirdly, embodiments of this application provide an electronic device, which includes:

[0015] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the panoramic video recording method provided in the first aspect of this application.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the panoramic video recording method provided in the first aspect of this application.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the panoramic video recording method provided in the first aspect of this application.

[0018] This application provides a method, apparatus, device, and storage medium for panoramic video recording. In real-time panoramic video recording, the anti-distortion camera image and spherical projection template at the current moment are first determined. Based on the camera's field of view, the mapping point texture of each spherical coordinate point within the spherical projection template in the anti-distortion camera image is determined. Then, based on the mapping point texture of each spherical coordinate point, pixel rendering is performed on the spherical projection template to obtain the panoramic video frame recorded at the current moment. This eliminates camera distortion and spherical projection distortion within the panoramic video frame, achieving distortion-free recording of panoramic video and improving the image quality. Furthermore, by using the camera's field of view to achieve convenient texture mapping from each spherical coordinate point to the anti-distortion camera image, no three-dimensional spatial processing of the spherical coordinate points is required. While ensuring the image quality of panoramic video recording, this further reduces frame processing overhead during panoramic video recording, thereby increasing the recording frame rate. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for panoramic video recording according to an embodiment of this application;

[0021] Figure 2a This is an exemplary schematic diagram of the spherical coordinate system in which the panoramic video is played in a spherical manner, as shown in the embodiments of this application.

[0022] Figure 2b This is an exemplary schematic diagram of the two-dimensional plane on which each panoramic video frame, after being recorded and stored, is projected using a spherical projection method, as shown in the embodiments of this application.

[0023] Figure 3 This is an exemplary schematic diagram of a spherical projection template shown in an embodiment of this application;

[0024] Figure 4 A flowchart illustrating another method for panoramic video recording as shown in an embodiment of this application;

[0025] Figure 5a This is a schematic diagram illustrating the imaging process of an anti-distortion camera image according to an embodiment of this application;

[0026] Figure 5b The spherical coordinates of any point in the field of view within a unit sphere in the spherical coordinate system shown in the embodiments of this application are... A schematic diagram showing the transformation relationship between the horizontal angle α_h and the vertical angle α_v between the point in the field of view and the camera optical axis;

[0027] Figure 6 A flowchart illustrating the method for determining the texture of mapping points of each spherical coordinate point within the spherical projection template provided in this application embodiment;

[0028] Figure 7a and Figure 7b These are exemplary schematic diagrams illustrating the classification principles of the first type of spherical coordinate points and the second type of spherical coordinate points within the spherical projection template provided in the embodiments of this application.

[0029] Figure 8 This is a schematic block diagram illustrating a panoramic video recording device according to an embodiment of this application;

[0030] Figure 9This is a schematic block diagram of an electronic device shown in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0033] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or solution described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0034] Considering that panoramic video frames recorded via spherical projection will exhibit projection distortion during the projection transformation process, and that lens manufacturing and assembly deviations will also result in some shooting distortion, various forms of distortion may exist within the panoramic video, this application provides a distortion-free panoramic video recording solution to eliminate these distortions. Based on the camera's field of view, the mapping point texture of each spherical coordinate point within the spherical projection template in the anti-distortion camera image at the current moment is determined. This texture is then used to perform pixel rendering on the spherical projection template, resulting in the recorded panoramic video frame at the current moment. This eliminates camera distortion and spherical projection distortion within the panoramic video frame, achieving distortion-free panoramic video recording and improving the image quality.

[0035] Figure 1This is a flowchart illustrating a method for panoramic video recording according to an embodiment of this application. This method can be executed by the panoramic video recording apparatus provided in this application, which can be implemented in any software and / or hardware manner. Exemplarily, the panoramic video recording apparatus can be applied to any electronic device, including but not limited to tablet computers, mobile phones (such as foldable phones, large-screen phones, etc.), wearable devices, in-vehicle devices, XR devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart TVs, smart screens, high-definition TVs, 4K TVs, smart speakers, smart projectors, etc. This application does not limit the specific type of electronic device.

[0036] Specifically, such as Figure 1 As shown, the method may include the following steps:

[0037] S110, determine the anti-distortion camera image and spherical projection template at the current moment.

[0038] To ensure the realism of panoramic video playback on other users' devices, spherical projection is usually used to record panoramic video. This transforms the real-time three-dimensional image of the scene into a two-dimensional plane for storage, resulting in panoramic video frames at each moment, thus achieving spherical playback of the panoramic video.

[0039] Therefore, it can be seen that each panoramic video frame recorded and stored can be a two-dimensional planar image projected using a spherical projection method. Furthermore, panoramic video is stereoscopic video and can be played through a corresponding spherical surface in a spherical coordinate system. For example... Figure 2a As shown, for any spherical point in spherical coordinates (that is, a pixel in a panoramic video frame), it can be determined by the azimuth angle. It is expressed as the elevation angle θ. Among them, the azimuth angle... The range of values ​​for can be [-π, π], and the range of values ​​for the elevation angle θ can be [-π / 2, π / 2].

[0040] Therefore, in order to ensure the rapid spherical rendering of panoramic video frames, this application can use the azimuth coordinates in a spherical coordinate system for the two-dimensional plane where each panoramic video frame is located. And the elevation coordinate θ, unfold the three-dimensional sphere according to The distributed two-dimensional plane serves as the two-dimensional plane in which the panoramic video frames reside, such as... Figure 2b As shown. Therefore, according to the spherical coordinates of each panoramic video frame... The distribution allows the panoramic video frame to be directly rendered onto a three-dimensional sphere in spherical coordinates, thus achieving a spherical presentation of the panoramic video frame.

[0041] In some possible implementations, the spherical projection method in this application can be equirectangular projection (ERP). ERP projection can determine the distance from any point on a three-dimensional unit sphere to a point according to... The projection mapping relationship on a two-dimensional plane of distribution.

[0042] In this application, since the two-dimensional plane containing each panoramic video frame in the panoramic video is based on the coordinates of each spherical point in the spherical coordinate system... The distribution is a two-dimensional plane. Therefore, in order to ensure the accurate recording of each panoramic video frame, this application can pre-set a spherical projection template according to the viewing angle of the panoramic video frame, so as to obtain panoramic video frames at different times by rendering different textures on the spherical projection template.

[0043] In this application, the rendering field of view of the spherical projection template is related to the camera's field of view. For example, assuming the camera's field of view is 180 degrees, then... Figure 3 As shown, the spherical projection template can be designed according to... The distributed rendering perspectives are all two-dimensional planar images in the range of [-π / 2, π / 2].

[0044] Furthermore, because the rendering texture of the spherical projection template facing each panoramic video frame is uncertain, the spherical projection template is arranged according to... Each coordinate point in the distribution can have the same grayscale value, allowing the spherical projection template to exist as a reference frame without containing any image information. For example, the spherical projection template in this application can be based on... A blank image where the gray value of each coordinate point is 255.

[0045] In order to perform corresponding texture rendering on the spherical projection template and obtain panoramic video frames at different times, this application can use the camera image at each time moment as the texture image of the spherical projection template at that time moment, so as to perform corresponding texture rendering on the spherical projection template.

[0046] Furthermore, during the real-time recording of panoramic video, the camera can capture images of the surrounding environment within the real scene in real time, which can then be used as camera images in this application. Therefore, the camera image at each moment will contain the texture information of the panoramic video frame at that moment. Thus, the camera image at each moment can be used as the texture image of the spherical projection template at that moment to obtain the panoramic video frame at that moment.

[0047] However, due to lens manufacturing and assembly deviations, real-time camera images captured in real-world scenes may exhibit some distortion. Therefore, to eliminate distortion within panoramic video frames, this application performs anti-distortion processing on the camera image at the current moment during the panoramic video recording process to obtain a corresponding anti-distortion camera image, which is then used as the texture image of the spherical projection template at the current moment.

[0048] As an optional implementation of this application, the anti-distortion camera image at the current moment can be determined by the following method: obtaining the original camera image at the current moment; using preset camera distortion parameters, performing anti-distortion processing on the original camera image to obtain the corresponding anti-distortion camera image.

[0049] In other words, during the real-time recording of panoramic video, if the original camera image at each moment is directly displayed on the screen, distortion will typically occur when the user views the displayed image through the lens installed in the XR device. Similarly, using the original camera image at each moment as the texture image of a spherical projection template to obtain the panoramic video frame at that moment will also introduce distortion into the panoramic video frame. Therefore, to eliminate shooting distortion within the panoramic video frame, this application employs an appropriate anti-distortion algorithm to perform anti-distortion processing on the original camera image each time it is acquired, resulting in an anti-distortion camera image for the current moment.

[0050] For example, assuming the original camera image has undergone distortion correction, when a user views the original camera image displayed on the screen through the lens installed in the XR device, they will typically see a camera image with barrel distortion. Therefore, by performing distortion correction on the original camera image, a camera image with pincushion distortion can be obtained, which serves as the corresponding distortion-corrected camera image. Then, when the user views the distortion-corrected camera image displayed on the screen through the lens installed in the XR device, they will see a distortion-free camera image.

[0051] S120: For each spherical coordinate point within the spherical projection template, determine the mapping point texture of the spherical coordinate point in the anti-distortion camera image based on the camera's field of view.

[0052] The pixel coordinates within an anti-distortion camera image represent the spatial points within the camera's field of view in the real scene, which, after being imaged, are distributed according to their spatial positions on the imaging plane, forming two-dimensional pixels. Similarly, the spherical coordinates within a spherical projection template represent the spatial points within the camera's field of view in the real scene, after being projected onto a spherical surface using a spherical projection method, and distributed according to their spherical coordinates. Two-dimensional pixels formed by the distribution.

[0053] Therefore, it can be seen that there is a corresponding mapping relationship between the pixel coordinates in the anti-distortion camera image and the spherical coordinates in the spherical projection template. The anti-distortion camera image contains texture information for each spatial point. Therefore, the anti-distortion camera image at each moment can be used as the texture image of the spherical projection template at that moment, and the specific texture of each spherical coordinate point can be determined by analyzing the coordinate mapping relationship from each spherical coordinate point in the spherical projection template to the coordinates in the anti-distortion camera image.

[0054] Analyzing the camera imaging principle reveals that the pixel coordinates of different spatial points within the imaging plane are related to the angle between that spatial point and the camera's optical axis. Furthermore, the angle between the four field-of-view boundary vertices and the camera's optical axis is half the camera's field-of-view angle, and after imaging, these angles are transformed into the four boundary vertices of the camera image. Since the pixel coordinates of the camera image within the imaging plane range from [0, 1], the pixel coordinates of the four boundary vertices of the camera image are known. Therefore, the pixel coordinates of any pixel within an anti-distortion camera image can be represented by the angle between that pixel and the camera's optical axis, and the camera's field-of-view angle.

[0055] Furthermore, in a three-dimensional coordinate system, the camera's optical axis can be taken as the corresponding X-axis. Therefore, by analyzing the angle between the spatial point corresponding to any pixel in the anti-distortion camera image and the X-axis, the angle between the spatial point corresponding to that pixel in the anti-distortion camera image and the camera's optical axis can be determined. Moreover, the spherical coordinates of the spatial point corresponding to any pixel in the anti-distortion camera image in the spherical coordinate system are represented as follows: The coordinates can be mapped to the spherical coordinates within the spherical projection template. The corresponding spherical coordinates. Therefore, the spherical coordinates of each spherical coordinate point within the spherical projection template. It can be represented by the angle between the spherical coordinate point and the X-axis (that is, the angle between the mapping point of the spherical coordinate point in the anti-distortion camera image and the camera optical axis).

[0056] Then, based on the relationship between the pixel coordinates of any pixel in the anti-distortion camera image, the angle between that pixel and the camera's optical axis, and the camera's field of view, and the spherical coordinates of each spherical coordinate point in the spherical projection template... By establishing the relationship between the spherical coordinates of this point and the angle between the mapped point of the point in the anti-distortion camera image and the camera's optical axis, the spherical coordinates of each point within the spherical projection template, represented by the camera's field of view, can be determined. The mapping relationship between the pixel coordinates of any pixel in the image from the anti-distortion camera.

[0057] Furthermore, after determining the anti-distortion camera image and the spherical projection template at the current moment, for each spherical coordinate point within the spherical projection template, based on the spherical coordinates of each spherical coordinate point within the spherical projection template represented by the camera's field of view angle, it is possible to... The mapping relationship between the pixel coordinates of any pixel in the image from the distortion correction camera can be established using the camera's field of view and the spherical coordinates of that spherical point. The pixel coordinates of the spherical coordinate point mapped to the anti-distortion camera image are determined, thereby determining the mapping point of the spherical coordinate point in the anti-distortion camera image. Then, the actual texture information of the mapping point is determined from the anti-distortion camera image, thus obtaining the texture of the mapping point of the spherical coordinate point in the anti-distortion camera image in this application.

[0058] In this application, the texture of the mapped point can be the pixel color value at the mapped point in the anti-distortion camera image.

[0059] S130: Based on the texture of the mapped points of each spherical coordinate point, the spherical projection template is pixel-rendered to obtain the panoramic video frame recorded at the current moment.

[0060] After determining the mapping point texture of each spherical coordinate point within the spherical projection template, the mapping point texture of each spherical coordinate point can be rendered onto the corresponding spherical coordinate point within the spherical projection template. This ensures that each spherical coordinate point within the spherical projection template has the same texture information as its mapping point in the anti-distortion camera image. Furthermore, the pixel-rendered spherical projection template is used as the panoramic video frame recorded at the current moment. Therefore, the panoramic video frames at each moment are arranged according to... When the panoramic video is played out in a spherical format, the projection distortion and shooting distortion caused by the camera during the spherical projection transformation of the panoramic video frames can be eliminated simultaneously, thereby improving the image quality of the panoramic video recording.

[0061] The technical solution provided in this application, under real-time recording of panoramic video, first determines the anti-distortion camera image and spherical projection template at the current moment. Based on the camera's field of view, the mapping point texture of each spherical coordinate point within the spherical projection template in the anti-distortion camera image is determined. Then, based on the mapping point texture of each spherical coordinate point, pixel rendering is performed on the spherical projection template to obtain the panoramic video frame recorded at the current moment, thereby eliminating camera distortion and spherical projection distortion within the panoramic video frame, achieving distortion-free recording of panoramic video, and improving the image quality of panoramic video recording. Furthermore, by using the camera's field of view to achieve convenient texture mapping from each spherical coordinate point to the anti-distortion camera image, no three-dimensional spatial processing of the spherical coordinate points is required. While ensuring the image quality of panoramic video recording, this further reduces the frame processing overhead during panoramic video recording, thereby increasing the recording frame rate of panoramic video.

[0062] As an optional implementation scheme in this application, in order to completely eliminate various distortions during panoramic video recording, this application needs to accurately analyze the mapping relationship between each spherical coordinate point in the spherical projection template and each pixel coordinate point in the anti-distortion camera image, in order to determine the true texture information of each spherical coordinate point. Therefore, this application can provide a detailed explanation of the specific mapping process from each spherical coordinate point in the spherical projection template to the anti-distortion camera image.

[0063] Figure 4 A flowchart illustrating another method for panoramic video recording as shown in an embodiment of this application. Figure 4 As shown, the method may specifically include the following steps:

[0064] S410, determine the anti-distortion camera image and spherical projection template at the current moment.

[0065] S420: For each spherical coordinate point within the spherical projection template, determine the mapped pixel coordinates of the spherical coordinate point based on the camera's field of view and the established spherical coordinate mapping relationship.

[0066] Analyzing the imaging process of distortion correction camera images reveals that each spatial point within the camera's field of view in the real scene, after being imaged by the camera, can be transformed into pixel coordinates within the distortion correction camera image according to their spatial distribution on the imaging plane. Furthermore, the imaging plane is perpendicular to the camera's optical axis, and the projection point of the camera's optical axis onto the imaging plane can be the origin of the pixel coordinate system within the imaging plane.

[0067] Therefore, the pixel coordinates of different spatial points in the imaging plane are related to the angle between that spatial point and the camera's optical axis. Furthermore, the angle between the four boundary vertices of the field of view (represented by the camera's field of view angle) and the camera's optical axis is half the camera's field of view angle, and after imaging, it transforms into the four boundary vertices in the distortion correction camera image. Since the pixel coordinates of the distortion correction camera image in the imaging plane range from [0, 1], the pixel coordinates of the four boundary vertices of the distortion correction camera image are known. Therefore, by analyzing the imaging process of any pixel in the distortion correction camera image, the pixel coordinate value of that pixel in the distortion correction camera image can be represented using the pixel coordinates of the four boundary vertices, the camera's field of view angle, and the angle between the corresponding spatial point and the camera's optical axis.

[0068] Furthermore, analyzing the spherical projection process of the spherical projection template reveals that each spatial point within the camera's field of view in the real scene, after undergoing spherical projection, can be represented according to spherical coordinates. The distribution is transformed into the coordinates of each spherical point within the spherical projection template.

[0069] In a three-dimensional coordinate system, with the camera's optical axis as the corresponding X-axis, the angle between the spatial point corresponding to any pixel in the distortion correction camera image and the X-axis can be analyzed to represent the angle between that spatial point and the camera's optical axis. Furthermore, the spherical coordinates of the spatial point corresponding to any pixel in the distortion correction camera image are represented in a spherical coordinate system as follows: The coordinates can be mapped to the spherical coordinates within the spherical projection template. The corresponding spherical coordinates.

[0070] Therefore, by analyzing the spherical coordinates of each spherical coordinate point within the spherical projection template... The relationship between the angle between the spatial point corresponding to the spherical coordinate point and the X-axis can be expressed using the angle between the spatial point corresponding to the spherical coordinate point and the X-axis.

[0071] Then, the calculation formula for the pixel coordinate value of any pixel in the anti-distortion camera image, represented by the pixel coordinates of the four boundary vertices in the anti-distortion camera image, the camera field of view, and the angle between the spatial point corresponding to that pixel and the camera optical axis, and the spherical coordinates of that spherical coordinate point, represented by the angle between the spatial point corresponding to each spherical coordinate point and the X-axis, are given. The calculation formulas are given above. This application allows for a comprehensive analysis of the two calculation formulas to obtain the spherical coordinates of each spherical coordinate point. The mapping relationship between the pixel coordinates of each pixel in the anti-distortion camera image is used as the spherical coordinate mapping relationship constructed in this application.

[0072] Furthermore, for each spherical coordinate point in the spherical projection template, this application can determine the spherical coordinates of that point according to the established spherical coordinate mapping relationship. By performing a unified analysis with the field of view of the camera used in this study, the pixel coordinates of the mapping point of the spherical coordinate point in the anti-distortion camera image are determined, thereby determining the mapping point of each spherical coordinate point in the anti-distortion camera image.

[0073] In some possible implementations, the spherical coordinate mapping relationship can be determined by the following steps:

[0074] The first step is to determine the corresponding pixel coordinate angle representation relationship based on the first pixel coordinates of the vertex of the field of view boundary in the imaging plane, which are represented by the camera's field of view angle, and the second pixel coordinates of any field of view spatial point in the imaging plane, which are represented by the angle between the field of view spatial point and the camera's optical axis.

[0075] By analyzing the camera imaging process of anti-distortion camera images, such as... Figure 5a As shown, based on the camera's field of view corresponding to the camera's field of view angle, four field-of-view boundary vertices can be determined. Furthermore, the camera's field of view angle can include both horizontal and vertical angles. After being imaged by the camera, the four field-of-view boundary vertices are transformed into the four boundary vertices within the anti-distortion camera image on the imaging plane. Since the camera's optical axis, acting as the X-axis of the three-dimensional spatial coordinate system, is perpendicular to the imaging plane, the projection point of the camera's optical axis onto the imaging plane can be used as the pixel origin of the pixel coordinate system within the imaging plane. Considering that the normalized uv coordinates of the pixels in the anti-distortion camera image on the imaging plane range from [0, 1], by taking the lower left corner of the imaging plane as the origin and translating the origin to the intersection of the camera's optical axis and the imaging plane (i.e., the center point of the imaging plane), the uv coordinates of the four boundary vertices within the anti-distortion camera image can be (-0.5, -0.5), (-0.5, 0.5), (0.5, -0.5), and (0.5, 0.5), respectively.

[0076] Therefore, since the horizontal angle between the vertex of the field of view boundary and the camera's optical axis is half of the camera's horizontal field of view, and the vertical angle between the vertex of the field of view boundary and the camera's optical axis is half of the camera's vertical field of view, the horizontal pixel coordinate value (i.e., the u value) of the vertex of the field of view boundary in the first pixel coordinate of the imaging plane can be: 0.5 = tan(Fov_H / 2) * f. The vertical pixel coordinate value (i.e., the v value) of the vertex of the field of view boundary in the first pixel coordinate of the imaging plane can be: 0.5 = tan(Fov_V / 2) * f.

[0077] Where Fov_H is the horizontal field of view of the camera, Fov_V is the vertical field of view of the camera, and f is the focal length of the camera.

[0078] For any point in the field of view within the camera's field of view, the horizontal and vertical angles between the point and the camera's optical axis can be used to analyze the horizontal pixel coordinates (i.e., the u-value) and vertical pixel coordinates (i.e., the v-value) of the point in the second pixel coordinates of the imaging plane, in the same way as described above.

[0079] For example, the horizontal pixel coordinate value (i.e., the u value) of any point in the field of view in the second pixel coordinates of the imaging plane can be: u = tan(α_h) * f. The vertical pixel coordinate value (i.e., the v value) of the point in the field of view in the second pixel coordinates of the imaging plane can be: v = tan(α_v) * f.

[0080] Where α_h is the horizontal angle between the point in the field of view and the optical axis of the camera, and α_v is the vertical angle between the point in the field of view and the optical axis of the camera.

[0081] Therefore, by comprehensively analyzing the first and second pixel coordinates, the corresponding pixel coordinate angle representation relationship can be obtained. This pixel coordinate angle representation relationship can be expressed as the calculation formula for the pixel coordinate value of any point in the field of view within the camera's field of view in the imaging plane (that is, the pixel coordinate value of any pixel in the anti-distortion camera image), expressed by the camera's field of view angle and the angle between the point in the field of view and the camera's optical axis.

[0082] Furthermore, the pixel coordinate angle representation can include both horizontal and vertical pixel coordinate angle representations. Specifically, the horizontal pixel coordinate angle representation can be: The relationship between the angles of the vertical pixel coordinates can be represented as follows:

[0083] Therefore, if we know the horizontal angle α_h and the vertical angle α_v between any point in the field of view and the camera's optical axis, we can determine the pixel coordinates of that point in the image of the anti-distortion camera.

[0084] The second step is to determine the corresponding spherical coordinate angle representation relationship based on the spherical coordinates of any point in the field of view and the angle between that point and the camera's optical axis.

[0085] Considering that any point in the camera's field of view can be transformed into a spherical coordinate point within a spherical projection template through spherical projection, then for any spherical coordinate point within the spherical projection template, it can be represented according to spherical coordinates. Transform it into a point in the field of view on any rendered sphere in spherical coordinates.

[0086] Taking the rendered sphere as an example, for any spherical coordinate point within the spherical projection template, the spherical coordinates... like Figure 5b As shown, the spherical coordinate point can be transformed into a field-of-view point on a unit sphere in spherical coordinate system.

[0087] At this time, as Figure 5b As shown, the field of view spatial points can be represented. The horizontal angle α_h and the vertical angle α_v between the X-axis and the X-axis (i.e., the camera's optical axis) can be used to obtain the spherical coordinates of a point in the field of view. There is a corresponding transformation relationship between the horizontal angle α_h and the vertical angle α_v between the point in the field of view and the camera optical axis.

[0088] Considering that during camera imaging, the camera's optical axis coincides with the X-axis and their positive directions are aligned, and that pixels located to the left of the camera's optical axis in the imaging plane have a negative angle with the optical axis, while pixels located to the right of the camera's optical axis have a positive angle with the optical axis, it can be concluded that for any spherical coordinate point within the spherical projection template... Azimuth coordinates in The horizontal angle α_h between the field of view point corresponding to this spherical coordinate point and the camera optical axis has the same value but opposite sign. That is, for any spherical coordinate point within the spherical projection template... It can be confirmed

[0089] Furthermore, for any spherical coordinate point within the spherical projection template The elevation angle coordinate θ in the figure, such as Figure 5b As shown, we can obtain PB = sinθ Therefore, based on the above relationship, it can be determined that

[0090] Therefore, the relationship between the included angles of the spherical coordinates can be determined using the above method. and

[0091] The third step is to determine the corresponding spherical coordinate mapping relationship based on the relationship between the pixel coordinate angle and the spherical coordinate angle.

[0092] By comprehensively analyzing the relationship between pixel coordinate angles and spherical coordinate angles, and removing the unknown horizontal angle α_h and vertical angle α_v between the field of view point and the camera optical axis, the mapping relationship between any spherical coordinate point in the spherical projection template and the spherical coordinate in the anti-distortion camera image can be obtained.

[0093] As can be seen from the above analysis, the spherical coordinate mapping relationship can include two types: the horizontal spherical coordinate mapping relationship and the vertical spherical coordinate mapping relationship. Specifically, the horizontal spherical coordinate mapping relationship can be... Spherical coordinate perpendicular mapping relationship

[0094] Furthermore, the above spherical coordinate mapping relationship is obtained by taking the center point of the imaging plane as the origin of the pixel coordinate system. However, in reality, the origin of the pixel coordinate system is usually the boundary vertex of the lower left corner of the imaging plane. Therefore, for the final spherical coordinate mapping relationship, it is necessary to add the offset of the origin to the above spherical coordinate mapping relationship.

[0095] In other words, the final spherical coordinate mapping relationship can be expressed as follows: Spherical coordinate perpendicular mapping relationship

[0096] Then, for each spherical coordinate point within the spherical projection template, the horizontal field of view (Fov_H) of the camera and the azimuth coordinates of that spherical coordinate point can be mapped according to the final spherical coordinate mapping relationship. Substituting these values ​​into the final spherical coordinate horizontal mapping formula yields the horizontal pixel coordinates of the mapped point on the spherical coordinate system. Furthermore, the camera's vertical field of view (Fov_V) and the azimuth coordinates of the spherical coordinate point can also be obtained. Substituting the elevation angle coordinate θ into the final spherical coordinate vertical mapping formula, we can obtain the vertical pixel coordinates of the mapped point of the spherical coordinate point.

[0097] Using the above method, the pixel coordinates of the mapping point of each spherical coordinate point in the spherical projection template within the anti-distortion camera image can be determined, thereby determining the mapping point of each spherical coordinate point within the anti-distortion camera image.

[0098] S430 determines the corresponding mapping point texture within the anti-distortion camera image based on the pixel coordinates of the mapping points.

[0099] After determining the pixel coordinates of the mapping point of each spherical coordinate point in the anti-distortion camera image, the corresponding mapping point texture can be determined in the anti-distortion camera image according to the pixel coordinates of the mapping point of each spherical coordinate point. This texture information is then used to render the texture of the spherical projection template.

[0100] S440 performs pixel rendering on the spherical projection template based on the texture of the mapped points of each spherical coordinate point to obtain the panoramic video frame recorded at the current moment.

[0101] The technical solution provided in this application, under real-time recording of panoramic video, first determines the anti-distortion camera image and spherical projection template at the current moment. Based on the camera's field of view, the mapping point texture of each spherical coordinate point within the spherical projection template in the anti-distortion camera image is determined. Then, based on the mapping point texture of each spherical coordinate point, pixel rendering is performed on the spherical projection template to obtain the panoramic video frame recorded at the current moment, thereby eliminating camera distortion and spherical projection distortion within the panoramic video frame, achieving distortion-free recording of panoramic video, and improving the image quality of panoramic video recording. Furthermore, by using the camera's field of view to achieve convenient texture mapping from each spherical coordinate point to the anti-distortion camera image, no three-dimensional spatial processing of the spherical coordinate points is required. While ensuring the image quality of panoramic video recording, this further reduces the frame processing overhead during panoramic video recording, thereby increasing the recording frame rate of panoramic video.

[0102] According to one or more embodiments of this application, in accordance with industry standards for spherical projection, the rendering field of view of the spherical projection template is usually set to 180 degrees or 360 degrees, so that different users can record panoramic videos in a unified format, enabling a user to successfully watch panoramic videos recorded by other users and achieving successful sharing of panoramic videos among different users.

[0103] However, since the maximum field of view of a camera is 180 degrees, this means that the maximum field of view for capturing an anti-distortion camera image can reach 180 degrees. Therefore, the rendering field of view of a spherical projection template is usually greater than or equal to the capturing field of view of the anti-distortion camera image. Thus, when the rendering field of view of the spherical projection template is greater than the capturing field of view of the anti-distortion camera image, only spherical coordinate points within the spherical projection template that are within the camera's field of view will have mapping points and corresponding texture information within the anti-distortion camera image. Conversely, spherical coordinate points within the spherical projection template that are outside the camera's field of view will not have mapping points and therefore no texture information within the anti-distortion camera image.

[0104] Therefore, when the rendering field of view of the spherical projection template is larger than the shooting field of view of the anti-distortion camera image, the anti-distortion camera image, used as the texture image of the spherical projection template, cannot fill the entire spherical projection template. In other words, after texture rendering, the spherical projection template will contain two parts: the textured area within the camera's field of view and the non-textured area outside the camera's field of view.

[0105] Since the number of pixels in an anti-distortion camera image is determined by the camera resolution, and the rendering field of view of a spherical projection template is larger than the shooting field of view of the anti-distortion camera image, if the entire spherical projection template is sampled using the camera resolution, the resolution of the texture region within the spherical projection template that is within the camera's field of view will be lower than the camera resolution. In other words, when analyzing the mapping points of each spherical coordinate point within the texture region of the spherical projection template that is within the camera's field of view to the texture points in the anti-distortion camera image, the resolution of the anti-distortion camera image during texture sampling will be reduced, resulting in a decrease in the sharpness of the spherical projection template after texture rendering.

[0106] For example, assuming a camera resolution of 3000*3000 and a camera field of view of 120 degrees, and a rendering field of view of 180 degrees for the spherical projection template, then if the entire spherical projection template is sampled using the camera resolution, the number of spherical coordinate points within the spherical projection template that fall within the camera's field of view is 3000*120 / 180 = 2000. In other words, the spherical projection template uses the anti-distortion camera image as a texture image, and the number of mapped points sampled within the anti-distortion camera image is 2000, making the resolution of the anti-distortion camera image during texture sampling 2000, which is smaller than the camera resolution.

[0107] To ensure the clarity of panoramic video recording, this application can provide a detailed description of the specific mapping process from each spherical coordinate point within the spherical projection template to the image from the anti-distortion camera.

[0108] Figure 6 A flowchart illustrating the method for determining the mapping point texture of each spherical coordinate point within the spherical projection template provided in this application embodiment. Figure 6 As shown, the method may specifically include the following steps:

[0109] S610 determines the optimal resolution of the spherical projection template based on the camera's field of view, camera resolution, and the rendering field of view of the spherical projection template.

[0110] Since the rendering field of view of a spherical projection template is usually larger than the camera's field of view, and only spherical coordinate points within the spherical projection template that are within the camera's field of view will have mapping points in the anti-distortion camera image, to ensure the clarity of panoramic video recording, the number of spherical coordinate points within the spherical projection template that are within the camera's field of view must be the same as the number of pixels in the anti-distortion camera image sampled at the camera resolution. In other words, the resolution of the texture area within the spherical projection template that is within the camera's field of view can be the same as the camera resolution.

[0111] Therefore, the ratio between the camera's field of view and the rendering field of view of the spherical projection template is equal to the ratio between the camera's resolution and the resolution of the entire spherical projection template. Based on this relationship, the resolution of the entire spherical projection template can be determined as its optimal resolution.

[0112] For example, assuming the camera resolution is 3000*3000 and the camera field of view is 120 degrees, and the rendering field of view of the spherical projection template is 180 degrees, then the optimal resolution of the spherical projection template can be 3000*180 / 120 = 4500.

[0113] S620, based on the optimal resolution, determines the first type of spherical coordinate points and the second type of spherical coordinate points within the spherical projection template.

[0114] After determining the optimal resolution of the spherical projection template, this optimal resolution can be used to sample the spherical projection template, obtaining the coordinates of each spherical point within the template. At this point, the number of spherical coordinate points within the camera's field of view is the same as the number of pixels in the anti-distortion camera image sampled at the camera resolution.

[0115] Since only spherical coordinate points within the camera's field of view within the spherical projection template have mapping points in the anti-distortion camera image, meaning they possess real texture information, this application, in order to improve the efficiency of determining the texture mapping between the spherical projection template and the anti-distortion camera image, can directly use spherical coordinate points within the camera's field of view within the spherical projection template as first-type spherical coordinate points, and use corresponding spherical coordinate points, including all spherical coordinate points within the spherical projection template outside the camera's field of view, as second-type spherical coordinate points.

[0116] Therefore, for each first type of spherical coordinate point, it is necessary to analyze the mapping point texture of the first type of spherical coordinate point in the anti-distortion camera image, without having to analyze the mapping of the second type of spherical coordinate points to the anti-distortion camera image. This greatly reduces the mapping calculation overhead between the spherical projection template and the anti-distortion camera image, and improves the efficiency of panoramic video recording.

[0117] S630, for each type of spherical coordinate point, determines the mapping point texture of the type of spherical coordinate point in the anti-distortion camera image based on the camera's field of view.

[0118] For each first-type spherical coordinate point, there exists a mapping point within the anti-distortion camera image. Therefore, based on the spherical coordinates of each spherical coordinate point within the spherical projection template represented by the camera's field of view... The mapping relationship between the pixel coordinates of any pixel point in the anti-distortion camera image and the spherical coordinate values ​​of the first type of spherical coordinate point can be utilized in this application. The pixel coordinates of the first type of spherical coordinate point mapped to the anti-distortion camera image are determined, thereby determining the mapping point of the first type of spherical coordinate point in the anti-distortion camera image. Then, the real texture information of the mapping point is determined from the anti-distortion camera image, thus obtaining the texture of the mapping point of the first type of spherical coordinate point in the anti-distortion camera image in this application.

[0119] S640: For each second type of spherical coordinate point, a preset texture is used as the mapping point texture of the second type of spherical coordinate point.

[0120] Since only the texture information of the first type of spherical coordinate points within the entire spherical projection template affects the realism of the panoramic video recording, while the texture information of the second type of spherical coordinate points does not affect the recording of the panoramic video, this application can pre-set a texture information according to the diverse presentation requirements of panoramic video. This preset texture can be a specific pixel color value, and this application does not limit the preset texture.

[0121] For each second type of spherical coordinate point, this application can directly assign a preset texture to each second type of spherical coordinate point to obtain the mapping point texture of the second type of spherical coordinate point.

[0122] The technical solution provided in this application determines the optimal resolution of the spherical projection template based on the camera's field of view, camera resolution, and the rendering field of view of the spherical projection template. Then, based on this optimal resolution, it identifies first-type spherical coordinate points that have mapping points within the anti-distortion camera image and second-type spherical coordinate points that do not. This allows for the analysis of the mapping point texture of the first-type spherical coordinate points within the anti-distortion camera image, without needing to analyze the mapping of the second-type spherical coordinate points to the anti-distortion camera image. This significantly reduces the mapping calculation overhead during panoramic video recording and improves the recording efficiency. Furthermore, the optimal resolution ensures that the number of first-type spherical coordinate points within the camera's field of view in the spherical projection template is the same as the number of pixels in the anti-distortion camera image after sampling at the camera resolution, avoiding downsampling during texture sampling in the anti-distortion camera image and thus guaranteeing the clarity of the panoramic video recording.

[0123] As an optional implementation in this application, the first type of spherical coordinate points and the second type of spherical coordinate points within the spherical projection template can be determined in two ways:

[0124] Method 1: Determine the spherical coordinate points within the spherical projection template based on its optimal resolution; select spherical coordinate points within the camera's field of view from the spherical coordinate points within the spherical projection template as the first type of spherical coordinate points; and select the remaining spherical coordinate points within the spherical projection template after selecting the first type of spherical coordinate points as the second type of spherical coordinate points.

[0125] In other words, by sampling the spherical projection template at its optimal resolution, we can obtain the coordinates of each spherical point within the template. Then, the number of spherical coordinates within the camera's field of view is the same as the number of pixels in the anti-distortion camera image after sampling at the camera resolution. This ensures that the texture area actually occupied by the spherical projection template after texture rendering has the same resolution as the anti-distortion camera image, avoiding downsampling of the anti-distortion camera image during texture sampling and guaranteeing the clarity of the panoramic video recording.

[0126] Furthermore, such as Figure 7a As shown, this application can select a portion of the spherical coordinate points within the camera's field of view from all the spherical coordinate points within the spherical projection template, as the first type of spherical coordinate points, in order to analyze the mapping from the first type of spherical coordinate points to the anti-distortion camera image. Furthermore, from all the spherical coordinate points within the spherical projection template, a portion of the remaining spherical coordinate points after selecting the first type of spherical coordinate points can be determined. These remaining spherical coordinate points are outside the camera's field of view within the spherical projection template and are thus designated as the second type of spherical coordinate points.

[0127] Method 2: Convert the spherical projection template into a first spherical projection template and a second spherical projection template; based on the image resolution of the first spherical projection template, use the spherical coordinate points within the first spherical projection template as first-type spherical coordinate points; based on the image resolution of the second spherical projection template, use the spherical coordinate points within the second spherical projection template as second-type spherical coordinate points.

[0128] In other words, after determining the optimal resolution of the spherical projection template, since this optimal resolution differs from the camera resolution, this application can convert the spherical projection template into two types—a first spherical projection template and a second spherical projection template—to ensure the resolution consistency between the spherical coordinate points within the spherical projection template and the mapping points of the anti-distortion camera image.

[0129] Specifically, the rendering field of view and image resolution of the first spherical projection template can be the camera's field of view and camera resolution, respectively, so that the texture mapping between the first spherical projection template and the anti-distortion camera image does not require downsampling. Moreover, the rendering field of view and image resolution of the second spherical projection template can be the rendering field of view and optimal resolution of the spherical projection template, respectively, in order to comply with the recording specifications of panoramic video.

[0130] For example, assuming the camera resolution is 3000*3000 and the camera field of view is 120 degrees, and the rendering field of view of the spherical projection template is 180 degrees, then the optimal resolution for the spherical projection template is 4500*4500. In this case, as... Figure 7b As shown, the spherical projection template can be converted into a first spherical projection template with a rendering field of view of 120 degrees and an image resolution of 3000*3000, and a second spherical projection template with a rendering field of view of 180 degrees and an image resolution of 4500*4500.

[0131] Then, by sampling the first spherical projection template using the camera resolution, spherical coordinate points within the first spherical projection template can be obtained as first-type spherical coordinate points, which are then used to determine the mapping point texture of the first-type spherical coordinate points in the anti-distortion camera image. Furthermore, by sampling the second spherical projection template using the optimal resolution, spherical coordinate points within the second spherical projection template can be obtained as second-type spherical coordinate points, which are then used to assign a preset texture to the second-type spherical coordinate points, thus obtaining the mapping point texture of the second-type spherical coordinate points.

[0132] It is understandable that after converting the spherical projection template into a first spherical projection template and a second projection template, the panoramic video frame obtained after texture rendering of the spherical projection template can be determined in the following way: according to the mapping point texture of each first type of spherical coordinate point, the first spherical projection template is pixel-rendered to obtain the first panoramic candidate frame at the current time; according to the mapping point texture of each second type of spherical coordinate point, the second spherical projection template is pixel-rendered to obtain the second panoramic candidate frame at the current time; the first panoramic candidate frame and the second panoramic candidate frame are merged to obtain the panoramic video frame recorded at the current time.

[0133] In other words, for the first spherical projection template, the mapping point texture of each first type of spherical coordinate point can be rendered to the corresponding spherical coordinate point in the first spherical projection template, so that each spherical coordinate point in the first spherical projection template can have the same texture information as the mapping point of the first type of spherical coordinate point in the anti-distortion camera image, thereby obtaining the first panoramic candidate frame at the current moment.

[0134] Furthermore, for the second spherical projection template, the mapping point texture of each second type of spherical coordinate point can be rendered to the corresponding spherical coordinate point within the second spherical projection template, so that each spherical coordinate point within the second spherical projection template can have a preset texture, thereby obtaining the second panoramic candidate frame at the current moment.

[0135] Then, as Figure 7b As shown, according to the spherical coordinate values ​​of the spherical coordinate points in the first and second panoramic candidate frames. The texture of the first panoramic candidate frame can be merged into the texture of the second panoramic candidate frame, so that the texture information of the first panoramic candidate frame is retained in the overlapping area, while the texture information of the second panoramic candidate frame is retained in the non-overlapping area, thus obtaining the panoramic video frame recorded at the current moment.

[0136] Figure 8 This is a schematic block diagram illustrating a panoramic video recording device according to an embodiment of this application. Figure 8 As shown, the device 800 may include:

[0137] Image determination module 810 is used to determine the anti-distortion camera image and spherical projection template at the current moment;

[0138] The texture mapping module 820 is used to determine the mapping point texture of each spherical coordinate point in the anti-distortion camera image based on the camera field of view angle for each spherical coordinate point in the spherical projection template.

[0139] The video frame recording module 830 is used to perform pixel rendering on the spherical projection template according to the mapping point texture of each spherical coordinate point to obtain the panoramic video frame recorded at the current moment.

[0140] In some embodiments of this application, the texture mapping module 820 may include:

[0141] The mapping point determination unit is used to determine the mapping point pixel coordinates of the spherical coordinate point based on the camera field of view and the established spherical coordinate mapping relationship;

[0142] The mapping point texture determination unit is used to determine the corresponding mapping point texture in the anti-distortion camera image based on the mapping point pixel coordinates.

[0143] In some embodiments of this application, the panoramic video recording device 800 may further include a mapping relationship determination module, through which the spherical coordinate mapping relationship is determined. This mapping relationship determination module may be used for:

[0144] Based on the first pixel coordinates of the vertex of the field of view boundary in the imaging plane, represented by the camera's field of view angle, and the second pixel coordinates of any field of view spatial point in the imaging plane, represented by the angle between the field of view spatial point and the camera's optical axis, determine the corresponding pixel coordinate angle representation relationship;

[0145] Based on the spherical coordinates of any point in the field of view and the angle between that point and the camera's optical axis, determine the corresponding spherical coordinate angle representation.

[0146] Based on the pixel coordinate angle representation relationship and the spherical coordinate angle representation relationship, the corresponding spherical coordinate mapping relationship is determined.

[0147] In some embodiments of this application, the spherical coordinate mapping relationship includes a horizontal spherical coordinate mapping relationship and a vertical spherical coordinate mapping relationship. The mapping point determination unit can be specifically used for:

[0148] Substituting the horizontal field of view of the camera and the azimuth coordinates of the spherical coordinate point into the horizontal mapping formula of the spherical coordinate point, the horizontal pixel coordinates of the mapped point of the spherical coordinate point are obtained;

[0149] Substituting the camera's vertical field of view, the azimuth coordinates and elevation coordinates of the spherical coordinate point into the spherical coordinate vertical mapping formula, we obtain the vertical pixel coordinates of the mapped point of the spherical coordinate point.

[0150] In some embodiments of this application, the image determination module 810 may be specifically used for:

[0151] Acquire the raw camera image at the current moment;

[0152] The original camera image is subjected to anti-distortion processing to obtain the corresponding anti-distortion camera image.

[0153] In some embodiments of this application, the texture mapping module 820 may further include:

[0154] The resolution optimization unit is used to determine the optimal resolution of the spherical projection template based on the camera's field of view, camera resolution, and the rendering field of view of the spherical projection template.

[0155] The coordinate point classification unit is used to determine the first type of spherical coordinate points and the second type of spherical coordinate points within the spherical projection template based on the optimal resolution.

[0156] The first texture determination unit is used to determine the mapping point texture of the first type of spherical coordinate point in the anti-distortion camera image for each first type of spherical coordinate point according to the camera field of view.

[0157] The second texture determination unit is used to use a preset texture as the mapping point texture of the second type of spherical coordinate point for each second type of spherical coordinate point;

[0158] Wherein, the first type of spherical coordinate point is a spherical coordinate point within the camera's field of view of the spherical projection template.

[0159] In some embodiments of this application, the coordinate point classification unit may be specifically used for:

[0160] Determine the spherical coordinate points within the spherical projection template based on the optimal resolution of the spherical projection template;

[0161] From the spherical coordinate points within the spherical projection template, select spherical coordinate points that are within the camera's field of view as the first type of spherical coordinate points;

[0162] The remaining spherical coordinate points after selecting the first type of spherical coordinate points within the spherical projection template are used as the second type of spherical coordinate points.

[0163] In some embodiments of this application, the coordinate point classification unit may also be specifically used for:

[0164] The spherical projection template is converted into a first spherical projection template and a second spherical projection template. The rendering field of view and image resolution of the first spherical projection template are the camera field of view and the camera resolution, respectively. The rendering field of view and image resolution of the second spherical projection template are the rendering field of view and the optimal resolution, respectively.

[0165] Based on the image resolution of the first spherical projection template, the spherical coordinate points within the first spherical projection template are taken as the first type of spherical coordinate points;

[0166] Based on the image resolution of the second spherical projection template, the spherical coordinate points within the second spherical projection template are used as the second type of spherical coordinate points.

[0167] In some embodiments of this application, the video frame recording module 830 may be specifically used for:

[0168] Based on the mapping point texture of each first type of spherical coordinate point, the first spherical projection template is pixel-rendered to obtain the first panoramic candidate frame at the current moment.

[0169] Based on the mapping point texture of each second type of spherical coordinate point, pixel rendering is performed on the second spherical projection template to obtain the second panoramic candidate frame at the current moment.

[0170] The first panoramic candidate frame and the second panoramic candidate frame are merged to obtain the panoramic video frame recorded at the current moment.

[0171] In this embodiment, during real-time recording of panoramic video, the anti-distortion camera image and spherical projection template at the current moment are first determined. Based on the camera's field of view, the mapping point texture of each spherical coordinate point within the spherical projection template in the anti-distortion camera image is determined. Then, based on the mapping point texture of each spherical coordinate point, pixel rendering is performed on the spherical projection template to obtain the panoramic video frame recorded at the current moment. This eliminates camera distortion and spherical projection distortion within the panoramic video frame, achieving distortion-free recording of panoramic video and improving the image quality. Furthermore, by using the camera's field of view to achieve convenient texture mapping from each spherical coordinate point to the anti-distortion camera image, no three-dimensional spatial processing of the spherical coordinate points is required. While ensuring the image quality of panoramic video recording, this further reduces the frame processing overhead during panoramic video recording, thereby increasing the recording frame rate.

[0172] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 8 The apparatus 800 shown can execute any of the method embodiments provided in this application, and the foregoing and other operations and / or functions of each module in the apparatus 800 are respectively for implementing the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0173] The apparatus 800 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0174] Figure 9 This is a schematic block diagram of an electronic device shown in an embodiment of this application.

[0175] like Figure 9 As shown, the electronic device 900 may include:

[0176] The system includes a memory 910 and a processor 920. The memory 910 stores computer programs and transfers the program code to the processor 920. In other words, the processor 920 can retrieve and run the computer program from the memory 910 to implement the methods described in the embodiments of this application.

[0177] For example, the processor 920 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0178] In some embodiments of this application, the processor 920 may include, but is not limited to:

[0179] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0180] In some embodiments of this application, the memory 910 includes, but is not limited to:

[0181] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0182] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 910 and executed by the processor 920 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0183] like Figure 9 As shown, the electronic device may also include:

[0184] Transceiver 930, which can be connected to processor 920 or memory 910.

[0185] The processor 920 can control the transceiver 930 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 930 may include a transmitter and a receiver. The transceiver 930 may further include antennas, and the number of antennas may be one or more.

[0186] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0187] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0188] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0189] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0191] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0192] The above-described embodiments are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for recording panoramic video, characterized in that, include: Determine the anti-distortion camera image and spherical projection template at the current moment. The spherical projection template is a two-dimensional planar image that does not contain any image information and is distributed according to the azimuth and elevation angles. For each spherical coordinate point within the spherical projection template, the mapping point texture of the spherical coordinate point in the anti-distortion camera image is determined according to the camera's field of view. The mapping relationship between the spherical projection template and the anti-distortion camera image is determined by the representation relationship between the pixel coordinates of any pixel in the anti-distortion camera image and the angle between that pixel and the camera's optical axis and the camera's field of view, and by the representation relationship between the spherical coordinates of each spherical coordinate point within the spherical projection template and the angle between the mapping point of that spherical coordinate point in the anti-distortion camera image and the camera's optical axis. Based on the texture of the mapped points of each spherical coordinate point, the spherical projection template is pixel-rendered to obtain the panoramic video frame recorded at the current moment.

2. The method according to claim 1, characterized in that, The step of determining the mapping point texture of the spherical coordinate points within the anti-distortion camera image based on the camera's field of view includes: Based on the camera field of view and the established spherical coordinate mapping relationship, determine the pixel coordinates of the mapped point of the spherical coordinate point; Based on the pixel coordinates of the mapped points, the corresponding mapped point texture is determined within the anti-distortion camera image.

3. The method according to claim 2, characterized in that, The spherical coordinate mapping relationship is determined in the following way: Based on the first pixel coordinates of the vertex of the field of view boundary in the imaging plane, represented by the camera's field of view angle, and the second pixel coordinates of any field of view spatial point in the imaging plane, represented by the angle between the field of view spatial point and the camera's optical axis, determine the corresponding pixel coordinate angle representation relationship; Based on the spherical coordinates of any point in the field of view and the angle between that point and the camera's optical axis, determine the corresponding spherical coordinate angle representation. Based on the pixel coordinate angle representation relationship and the spherical coordinate angle representation relationship, the corresponding spherical coordinate mapping relationship is determined.

4. The method according to claim 2, characterized in that, The spherical coordinate mapping relationship includes a horizontal spherical coordinate mapping formula and a vertical spherical coordinate mapping formula. Determining the mapped pixel coordinates of the spherical coordinate points based on the camera field of view and the established spherical coordinate mapping relationship includes: Substituting the horizontal field of view of the camera and the azimuth coordinates of the spherical coordinate point into the horizontal mapping formula of the spherical coordinate point, the horizontal pixel coordinates of the mapped point of the spherical coordinate point are obtained; Substituting the camera's vertical field of view, the azimuth coordinates and elevation coordinates of the spherical coordinate point into the spherical coordinate vertical mapping formula, we obtain the vertical pixel coordinates of the mapped point of the spherical coordinate point.

5. The method according to claim 1, characterized in that, Determine the distortion-reduced camera image at the current moment, including: Acquire the raw camera image at the current moment; The original camera image is subjected to anti-distortion processing to obtain the corresponding anti-distortion camera image.

6. The method according to claim 1, characterized in that, The step of determining the mapping point texture of each spherical coordinate point within the spherical projection template in the anti-distortion camera image based on the camera's field of view includes: The optimal resolution of the spherical projection template is determined based on the camera's field of view, camera resolution, and the rendering field of view of the spherical projection template. Based on the optimal resolution, determine the first type of spherical coordinate points and the second type of spherical coordinate points within the spherical projection template; For each of the first type of spherical coordinate points, the mapping point texture of the first type of spherical coordinate points in the anti-distortion camera image is determined according to the camera field of view. For each second type of spherical coordinate point, a preset texture is used as the mapping point texture of the second type of spherical coordinate point; Wherein, the first type of spherical coordinate point is a spherical coordinate point within the camera's field of view of the spherical projection template.

7. The method according to claim 6, characterized in that, The step of determining the first type of spherical coordinate points and the second type of spherical coordinate points within the spherical projection template based on the optimal resolution includes: Determine the spherical coordinate points within the spherical projection template based on the optimal resolution of the spherical projection template; From the spherical coordinate points within the spherical projection template, select spherical coordinate points that are within the camera's field of view as the first type of spherical coordinate points; The remaining spherical coordinate points after selecting the first type of spherical coordinate points within the spherical projection template are used as the second type of spherical coordinate points.

8. The method according to claim 6, characterized in that, The step of determining the first type of spherical coordinate points and the second type of spherical coordinate points within the spherical projection template based on the optimal resolution further includes: The spherical projection template is converted into a first spherical projection template and a second spherical projection template. The rendering field of view and image resolution of the first spherical projection template are the camera field of view and the camera resolution, respectively. The rendering field of view and image resolution of the second spherical projection template are the rendering field of view and the optimal resolution, respectively. Based on the image resolution of the first spherical projection template, the spherical coordinate points within the first spherical projection template are taken as the first type of spherical coordinate points; Based on the image resolution of the second spherical projection template, the spherical coordinate points within the second spherical projection template are used as the second type of spherical coordinate points.

9. The method according to claim 8, characterized in that, The step of rendering pixels on the spherical projection template based on the mapping point texture of each spherical coordinate point to obtain the panoramic video frame recorded at the current moment includes: Based on the mapping point texture of each first type of spherical coordinate point, the first spherical projection template is pixel-rendered to obtain the first panoramic candidate frame at the current moment. Based on the mapping point texture of each second type of spherical coordinate point, pixel rendering is performed on the second spherical projection template to obtain the second panoramic candidate frame at the current moment. The first panoramic candidate frame and the second panoramic candidate frame are merged to obtain the panoramic video frame recorded at the current moment.

10. A device for panoramic video recording, characterized in that, include: The image determination module is used to determine the anti-distortion camera image and the spherical projection template at the current moment. The spherical projection template is a two-dimensional planar image that does not contain any image information and is distributed according to the azimuth and elevation angles. The texture mapping module is used to determine the texture of the mapping point in the anti-distortion camera image for each spherical coordinate point in the spherical projection template, based on the camera's field of view. The mapping relationship between the spherical projection template and the anti-distortion camera image is determined by the relationship between the pixel coordinates of any pixel in the anti-distortion camera image and the angle between that pixel and the camera's optical axis and the camera's field of view, and by the relationship between the spherical coordinates of each spherical coordinate point in the spherical projection template and the angle between the mapping point of that spherical coordinate point in the anti-distortion camera image and the camera's optical axis. The video frame recording module is used to perform pixel rendering on the spherical projection template according to the mapping point texture of each spherical coordinate point to obtain the panoramic video frame recorded at the current moment.

11. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the panoramic video recording method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method of panoramic video recording as described in any one of claims 1-9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method for panoramic video recording as described in any one of claims 1-9.

Images