Panoramic VR pole projection correction method, device, equipment and medium

Abstract

The invention relates to the technical field of VR projection, in particular to a panoramic VR pole projection correction method, device and equipment and a medium. The purpose of gradually compressing the display size of the two-pole low-information-entropy regions is achieved by actively introducing slight deformation along with the opportunity that the user field angle rotates towards the two-pole regions, that is, when the user field angle is at the initial horizontal position, the original picture proportion is kept without additional deformation; and when the user field angle is turned to the upper and lower vertex positions, the deformation of the pole region is maximum. The deformation process and the rotation of the FOV are kept synchronous, that is, the compression of the two-pole region is smaller when the view angle is rotated towards the two poles, and meanwhile, the introduction of the deformation is gradually increased, so that the phenomenon that the impression is abnormal due to non-transition deformation is avoided. The problem that an existing panorama is blurred in upper and lower vertex areas and is displayed abnormally in a concentric circle or vortex form is solved. The method is easy and convenient to process, small in calculation amount and more suitable for equipment with limited calculation performance such as embedded equipment.

Description

Technical Field

[0001] The present disclosure relates to the field of VR projection technology, and in particular to a method, device, equipment and medium for panoramic VR pole projection correction. Background Art

[0002] VR panoramic playback technology is a technology that uses computer image technology to simulate a virtual three-dimensional world, allowing users to achieve all-round and no-dead-angle observation and interaction in this world. It combines panoramic image technology and virtual reality technology to provide users with an immersive visual experience. The technologies behind it mainly include:

[0003] Multi-angle shooting: Use professional cameras to shoot from multiple angles at the same location to ensure that every angle of the scene is captured.

[0004] Image stitching: Use image processing software to accurately stitch and fuse the captured photos to form a seamless panoramic image.

[0005] Virtual reality technology: Generate a three-dimensional virtual environment through a computer, use VR equipment to interact with the virtual environment, and simulate human vision, hearing and other senses. Specifically, it uses VR panoramic playback technology, hardware devices such as panoramic cameras, combined with image processing algorithms, to render and present VR projection videos on virtual reality terminals such as VR glasses and set-top boxes. The most commonly used projection model in panoramic VR playback is the VR sphere model.

[0006] VR sphere model usually refers to a model that uses equidistant cylindrical projection (ERP, Equi-Rectangular Projection) (also known as longitude and latitude mapping), which is the most common projection method of the world map. It uses the Mercator projection rule, which is to project the longitude and latitude lines equidistantly (or densely) onto a rectangular plane. The advantage of this format is that it is more intuitive and the projection is rectangular. The disadvantages are also obvious. The number of pixels projected from the upper and lower poles of the sphere is large, while the number of pixels in the equatorial area with richer details is relatively small, resulting in poor clarity when restored. In addition, the picture in this format is obviously distorted when it is not rendered.

[0007] With the continuous advancement of technology, VR panoramic playback technology is not only providing an immersive experience, but also expanding its application areas and improving user experience. The problem of poor projection clarity in the upper and lower vertex areas of the original ERP projection method has gradually become sensitive.

[0008] In summary, there is an urgent need for a method to improve the display effect of polar regions based on VR panoramic models. Summary of the invention

[0009] In view of the above problems, the present disclosure provides a method, device, equipment and medium for panoramic VR pole projection correction, which adjusts the projection rules of the original ERP model in the area near the pole of the VR ERP model, and cuts or reduces the area with low original pole information entropy, thereby achieving the purpose of improving the display effect.

[0010] In a first aspect, a method for panoramic VR pole projection correction is provided, the method comprising:

[0011] Divide the original video image into a polar region and a central region, and determine the range of the polar region and the central region;

[0012] Based on the two polar regions, the phased deformation intervals and corresponding deformation ratios are determined;

[0013] The polar regions and the central region of the image are scaled and spliced ​​according to the FOV angle to obtain a corrected image.

[0014] Furthermore, the scope of the polar and central regions is determined, including:

[0015] The width of the polar regions and the central region is consistent and equal to the original video width;

[0016] The heights of the upper and lower polar regions are preset proportions of the original video height, and the remainder of the original video height minus the heights of the upper and lower polar regions is the height of the central region; wherein the preset proportions can be adjusted according to the actual scene, and the value range is (0, 6 / 180].

[0017] Furthermore, the phased deformation intervals and corresponding deformation ratios are determined, including:

[0018] Determine the number of deformation intervals n, and divide the upper hemisphere and the lower hemisphere into n intervals;

[0019] Among the n intervals, the initial interval in the horizontal position is not compressed, and the deformation ratio between each interval increases by 1 / 180 from the initial interval to the two poles.

[0020] Furthermore, the number of deformation intervals n is determined, including:

[0021] The number of deformation intervals n is determined by dividing the height of the two polar regions by the number of integer ratios of M.

[0022] Furthermore, the polar regions and the central region of the image are scaled and stitched according to the FOV angle, including:

[0023] Calculate the angle γ between the center point of the FOV angle of the image to be refreshed and the X-axis and Z-axis planes in the Cartesian coordinate system. By comparing the intervals, determine the compression ratio of the polar regions and perform compression.

[0024] Further, the compression ratio of the selected polar regions is determined and compressed, including:

[0025] Map γ to the deformation interval of [0°, 90°), and obtain the corresponding compression ratio a% according to the deformation interval; 0° is the angle between the X-axis and Z-axis planes;

[0026] Assume the width of the original video image is W and the height is H, the initial height of the polar regions is A, the height of the central region is H–2A, and the height of the central region is recorded as B;

[0027] The deformation ratio of the polar regions retained in the current interval is a%, then the corrected image is composed of three parts. The content of the W*B in the central area of ​​the original video is stretched to W*H*(1-2a%), and then the 2 W*A contents at the two poles of the original video are spliced ​​and reduced to an image of W*H*a%. The splicing order is consistent with the arrangement order of the original video.

[0028] Furthermore, it also includes:

[0029] The corrected image is sent to the ERP projection matrix for projection transformation, and VR orthogonal projection rendering is applied to display the content of the FOV perspective.

[0030] In a second aspect, a device for panoramic VR pole projection correction includes: a region division unit, a deformation ratio confirmation unit, and a correction unit;

[0031] A region division unit, used for dividing the original video image into a polar region and a central region, and determining the range of the polar region and the central region;

[0032] A deformation ratio confirmation unit, used to determine the phased deformation intervals and corresponding deformation ratios based on the two polar regions;

[0033] The correction unit is used to scale and splice the polar regions and the central region of the image according to the FOV angle to obtain a corrected image.

[0034] In a third aspect, an electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0035] a memory storing a computer program;

[0036] The processor is used to implement the above-mentioned method for panoramic VR pole projection correction when executing the computer program stored in the memory.

[0037] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for panoramic VR pole projection correction.

[0038] The present disclosure has at least the following beneficial effects:

[0039] The present invention achieves the purpose of gradually compressing the display size of the low information entropy areas at the poles by actively introducing slight deformation as the user's field of view rotates toward the poles. That is, when the user's field of view is in the initial horizontal position, the original screen ratio is maintained without additional deformation; when the user's field of view turns to the upper and lower vertex positions, the pole area is deformed the most, and the pole is compressed to the minimum at this time. The deformation process needs to be synchronized with the rotation of the FOV, that is, the more the viewing angle rotates toward the poles, the less the pole area is compressed. At the same time, the introduction of deformation is gradually increased to avoid deformation without transition, which causes abnormal viewing experience.

[0040] The present invention can improve the display abnormality problem that the upper and lower vertex areas are blurred and appear as concentric circles or vortexes when the existing panoramic VR uses the ERP model for projection. At the same time, the present invention is simple to process, has a small amount of calculation, and is more suitable for devices with limited computing performance such as embedded devices.

[0041] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a flowchart of a correction method according to an embodiment of the present disclosure;

[0044] Figure 2 This is a schematic diagram of the structure of the correction device according to an embodiment of the present disclosure;

[0045] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure;

[0046] Figure 4 This is a schematic diagram of the area division of an embodiment of the present disclosure;

[0047] Figure 5 This is a schematic diagram of the comparison effect before and after the implementation of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0049] like Figure 1 As shown, a method for correcting panoramic VR pole projection includes:

[0050] S101, dividing the original video image into a polar region and a central region, and determining the range of the polar region and the central region;

[0051] S102, determining the phased deformation intervals and corresponding deformation ratios based on the two polar regions;

[0052] S103, scaling and splicing the polar regions and the central region of the image according to the FOV angle to obtain a corrected image.

[0053] The specific implementation is as follows:

[0054] As the user's field of view rotates toward the two polar regions, slight deformation is actively introduced to gradually compress the display size of the two polar low information entropy regions. That is, when the user's field of view is in the initial horizontal position, the original picture ratio is maintained without additional deformation; when the user's field of view turns to the upper and lower vertex positions, the pole area is deformed the most, and the pole is compressed to the minimum at this time.

[0055] This deformation process needs to be synchronized with the rotation of FOV, that is, the further the viewing angle rotates toward the poles, the smaller the polar regions are compressed. At the same time, the deformation introduced is gradually increased to avoid deformation without transition, which would cause abnormal viewing experience.

[0056] The key features of the whole process are:

[0057] On the VR terminal, as the FOV angle is adjusted, the proportion of the pole information in the projection is changed during the angle adjustment stage;

[0058] The process of correcting the introduced deformation is completed by pre-scaling and cropping, and then applying the projection model.

[0059] The whole process does not require additional model modification and the calculation is fast.

[0060] The specific steps include:

[0061] S101, determine the range of the polar regions and the central region. When the player of the VR terminal is started, the original video image is divided into the polar regions and the central region, and the proportion of each region is confirmed according to the configuration. The width of the polar regions and the central region is the same, which is equal to the width of the original video. The height is set by default. The height of the upper and lower polar regions is a preset ratio of the original video height. The preset ratio defaults to 1 / 36 (ie 5 / 180). The height of the original video minus the height of the upper and lower polar regions is the height of the central region (34 / 36 of the original video height). This preset ratio can be configured and modified, allowing fine-tuning according to the actual application scenario. The value range is (0, 6 / 180].

[0062] S102, determine the interval division of the staged deformation and the corresponding deformation ratio. Determine the number of deformation intervals n by dividing the height of the two polar regions by the number of integers of 1 / 180, and then divide the upper hemisphere and the lower hemisphere into n intervals of 90° each. Between intervals, the initial interval is not compressed, and the height of the two polar regions is compressed incrementally by 1 / 180.

[0063] S103, scale and splice the polar regions and the central region of the image according to the FOV angle. Calculate the size of the angle between the center point of the FOV angle of the image to be refreshed and the X-axis and Z-axis planes in the Cartesian coordinate system, and determine the compression ratio that should be selected for the polar regions by comparing the intervals. Suppose the width W and height H of the original video image, the initial height of the polar regions is A, and the height of the central region is H-2A, recorded as B. The deformation ratio of the polar regions retained in the current interval is a%, then the new image is composed of three parts, the content of the W*B in the central area of ​​the original video is stretched to W*(H*(1-2a%)), and then the contents of the 2 W*A at the two poles of the original video are spliced ​​to reduce the image to W*(H*a%), and the splicing order is consistent with the arrangement order of the original video.

[0064] S104, the new image is subjected to ERP projection transformation. The newly scaled and spliced ​​image is sent to the ERP projection matrix for projection transformation, and VR orthogonal projection rendering is applied to display the content of the FOV perspective.

[0065] like Figure 2 As shown, a device for panoramic VR pole projection correction includes: a region division unit 201, a deformation ratio confirmation unit 202 and a correction unit 203;

[0066] The region division unit 201 is used to divide the original video image into a polar region and a central region, and determine the range of the polar region and the central region;

[0067] A deformation ratio confirmation unit 202, used to determine the phased deformation intervals and corresponding deformation ratios based on the two polar regions;

[0068] The correction unit 203 is used to scale and splice the polar regions and the central region of the image according to the FOV angle to obtain a corrected image.

[0069] like Figure 3 As shown, the present disclosure provides an electronic device, including a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302 and the memory 303 communicate with each other through the communication bus 304;

[0070] Memory 303, storing computer programs;

[0071] The processor 301 is configured to implement the above method when executing the computer program stored in the memory 303 .

[0072] The present disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0073] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0074] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0075] In order to enable those skilled in the art to better understand the present disclosure, the principles of the present disclosure are described as follows in conjunction with the accompanying drawings:

[0076] Confirm the range of the polar regions and the central region. For example, the default configuration is used, that is, the upper and lower polar regions are equally divided into 5 / 180, and the rest is the central region.

[0077] Determine the relationship between the degree of deformation and the distribution of γ. Figure 4As shown in the figure, the angle between the center point P of the FOV field of view and the X-axis and Z-axis planes in the Cartesian coordinate system is denoted as γ. According to the original height of each pole of each deformation of 1 / 180, it is necessary to adjust and divide it into 4 deformation stages in the upper and lower hemispheres. Taking P in the upper hemisphere as an example, γ is mapped to the interval [0°, 90°), where 0° is the angle between the X-axis and Z-axis planes. The deformation interval is divided as follows: [0°, 16°), [16°, 32°), [32°, 48°), [48°, 64°), [64°, 90°), corresponding to the upper and lower pole areas of 5 / 180 (no deformation), 4 / 180, 3 / 180, 2 / 180, and 1 / 180 respectively. When γ points to the lower hemisphere, γ is mapped to the interval [0°, -90°), and the deformation interval is divided into [0°, -16°),

[0078] [-16°,-32°), [-32°,-48°), [-48°,-64°), [-64°,-90°), corresponding to retaining the upper and lower pole areas at 5 / 180 (no deformation), 4 / 180, 3 / 180, 2 / 180, and 1 / 180 respectively.

[0079] Calculate the γ angle after the next rotation, and apply different deformation ratios when γ falls in different intervals. For example, in the current example, the original video has a resolution of 3840*2160, corresponding to W=3840, H=2160, and A=2160*5 / 180=60 in step S103. By executing step 1, the original central area is 3840*2040 (H-2A, that is, 2040=2160-60*2), and the polar areas are 3840*60 respectively. When γ is 50°, it is necessary to compress the polar areas to 2 / 180, and stretch the central area according to the remaining height size, that is, the polar areas are scaled to 3840*24 in the north and south (the new heights of the polar areas are H*a%, that is, 24=2160*2 / 180), and the content of the central area is scaled to 3840*2112 (the new height of the central area is H*(1-2a%), that is, 2112=2160*(1-2 / 180)), and they are stitched into a new video image.

[0080] The new image after stitching is subjected to ERP projection. Figure 5 It is a schematic diagram for comparing a video image before and after the implementation of the present disclosure.

[0081] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for panoramic VR pole projection correction, characterized in that: The method comprises: Divide the original video image into a polar region and a central region, and determine the range of the polar region and the central region; Based on the two polar regions, the phased deformation intervals and corresponding deformation ratios are determined; The polar regions and the central region of the image are scaled and spliced ​​according to the FOV angle to obtain a corrected image.

2. The method for correcting panoramic VR pole projection according to claim 1, characterized in that: Determine the scope of the polar and central regions, including: The width of the polar regions and the central region is consistent and equal to the original video width; The heights of the upper and lower polar regions are preset proportions of the original video height, and the remainder of the original video height minus the heights of the upper and lower polar regions is the height of the central region; wherein the preset proportions can be adjusted according to the actual scene, and the value range is (0, 6 / 180].

3. The method for correcting panoramic VR pole projection according to claim 1, characterized in that: Determine the phased deformation intervals and corresponding deformation ratios, including: Determine the number of deformation intervals n, and divide the upper hemisphere and the lower hemisphere into n intervals; Among the n intervals, the initial interval in the horizontal position is not compressed, and the deformation ratio between each interval increases by 1 / 180 from the initial interval to the two poles.

4. The method for correcting panoramic VR pole projection according to claim 3, characterized in that: Determine the number of deformation intervals n, including: The number of deformation intervals n is determined by dividing the height of the two polar regions by the number of integer ratios of M.

5. The method for correcting panoramic VR pole projection according to claim 1, characterized in that: Scaling and stitching of the polar and central areas of the image according to the FOV angle, including: Calculate the angle γ between the center point of the FOV angle of the image to be refreshed and the X-axis and Z-axis planes in the Cartesian coordinate system. By comparing the intervals, determine the compression ratio of the polar regions and perform compression.

6. A method for correcting panoramic VR pole projection according to claim 5, characterized in that: Determine the compression ratio of the polar region selection and perform compression, including: Map γ to the deformation interval of [0°, 90°), and obtain the corresponding compression ratio a% according to the deformation interval; 0° is the angle between the X-axis and Z-axis planes; Assume the width of the original video image is W and the height is H, the initial height of the polar regions is A, the height of the central region is H–2A, and the height of the central region is recorded as B; The deformation ratio of the polar regions retained in the current interval is a%, then the corrected image is composed of three parts. The content of the W*B in the central area of ​​the original video is stretched to W*H*(1-2a%), and then the 2 W*A contents at the two poles of the original video are spliced ​​and reduced to an image of W*H*a%. The splicing order is consistent with the arrangement order of the original video.

7. The method for correcting panoramic VR pole projection according to claim 1, characterized in that: Also includes: The corrected image is sent to the ERP projection matrix for projection transformation, and VR orthogonal projection rendering is applied to display the content of the FOV perspective.

8. A device for panoramic VR pole projection correction, characterized in that: include: Area division unit, deformation ratio confirmation unit and correction unit; A region division unit, used for dividing the original video image into a polar region and a central region, and determining the range of the polar region and the central region; A deformation ratio confirmation unit, used to determine the phased deformation intervals and corresponding deformation ratios based on the two polar regions; The correction unit is used to scale and splice the polar regions and the central region of the image according to the FOV angle to obtain a corrected image.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a memory storing a computer program; The processor is used to implement a method for correcting a panoramic VR pole projection as described in any one of claims 1 to 7 when executing a computer program stored in a memory.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for correcting panoramic VR pole projection according to any one of claims 1 to 7 is implemented.

Images