Panoramic video free view angle display method and device, terminal and storage medium

Abstract

The invention provides a panoramic video free view angle display method and device, a terminal and a storage medium, and the method comprises the steps: obtaining a geometric projection body constructed for a target real scene in a virtual scene, and obtaining a panoramic video of the target real scene shot by a plurality of cameras; projecting the panoramic video to a geometric projection body to obtain a virtual space of panoramic image projection; based on the real-time viewpoint of the user and the real-time sight direction of the user, correction projection from the camera of the target real scene to the real-time viewpoint of the user is completed in the virtual space; and performing scene rendering on the virtual space after the projection correction and displaying a rendered picture. According to the invention, the correction projection of the real-time viewpoint of the user is carried out in the virtual space which is provided with the geometric projection body and has panoramic image projection, and the selectable viewpoint position of the user is not restricted by the layout of the camera, so that the user can freely move and watch in the virtual space, and the accuracy and continuity of scene space perception are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a panoramic video free-viewing-angle display method, device, terminal and storage medium. Background Art

[0002] At present, there are two mainstream methods for displaying panoramic videos of large scenes: the first method is to directly splice the picture information of all panoramic videos on a large screen and display them all on the screen; the second method is to make the user's perspective jump to the position of the monocular camera or panoramic camera in the scene, that is, to place the user's viewpoint at the position of the camera in the scene, and watch the video content at a specific location in combination with a screen or VR glasses (head-mounted display).

[0003] The advantage of the first method is that it can intuitively display all the information in the scene, but the displayed information is limited by the screen size and lacks an immersive sense of presence in the scene; the advantage of the second method is that it can provide an immersive presence experience, which allows users to judge the on-site situation more intuitively in remote operation and monitoring application scenarios. The disadvantage is that the selectable user viewpoint position is constrained by the camera layout and cannot move freely in the scene.

[0004] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, device, terminal and storage medium for free viewing angle display of panoramic video are provided, aiming to solve the problem in the prior art that the selectable user viewpoint position is constrained by the camera layout and cannot move freely within the scene.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] A method for displaying a panoramic video with free viewing angle, wherein the method comprises:

[0008] Acquire a geometric projection body constructed for a target real scene in a virtual scene, and acquire a panoramic video of the target real scene captured by a plurality of cameras;

[0009] Projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection;

[0010] Acquire a user's real-time viewpoint and a user's real-time sight direction, and complete a correction projection from a camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction;

[0011] The virtual space after the corrected projection is subjected to scene rendering and the rendered image is displayed.

[0012] In one embodiment of the present application, the shape of the geometric projection body matches the shape of the target real scene.

[0013] In one embodiment of the present application, before projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection, the method further includes:

[0014] Determine the resolution of the panoramic video, and search a pre-stored resolution-size correspondence relationship according to the resolution to obtain a virtual scene size corresponding to the resolution;

[0015] The scale of the geometric projection volume is adjusted in the virtual scene based on the virtual scene size of the geometric projection volume.

[0016] In one embodiment of the present application, projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection includes:

[0017] Based on actual installation positions of a plurality of cameras in the target real scene, determining a virtual installation position of a virtual camera corresponding to each camera in the geometric projection body;

[0018] Obtain camera area images captured by a plurality of cameras in the target real scene according to the panoramic video, and obtain display images corresponding to each of the virtual cameras based on the virtual installation positions of the virtual cameras and the camera area images;

[0019] A virtual space for panoramic image projection is obtained based on the display screens corresponding to the virtual cameras.

[0020] In one embodiment of the present application, obtaining a user's real-time viewpoint and a user's real-time sight direction, and completing a correction projection from a camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction, includes:

[0021] Acquire the user's real-time viewpoint and the user's real-time sight direction, determine the real-time viewpoint position of the user's real-time viewpoint in the virtual space, and assume the real-time viewpoint position as a viewpoint virtual camera;

[0022] Determine a target virtual camera corresponding to the viewpoint virtual camera in the real-time sight line direction of the user, and determine a target camera corresponding to the target virtual camera in a target real scene;

[0023] A corrective projection from the target virtual camera to the viewpoint virtual camera is completed based on the camera parameters of the target camera.

[0024] In one embodiment of the present application, completing the correction projection from the target virtual camera to the viewpoint virtual camera based on the camera parameters of the target camera includes:

[0025] Determine, based on the camera parameters of the target camera, a homography matrix between pixels photographed by the target virtual camera and pixels photographed by the viewpoint virtual camera;

[0026] A corrective projection from the target virtual camera to the viewpoint virtual camera is completed based on the homography matrix.

[0027] In one embodiment of the present application, obtaining the user's real-time viewpoint and the user's real-time sight direction, and before completing the correction projection from the camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction, further includes:

[0028] A user viewpoint selection range area is pre-divided in the virtual space.

[0029] The present application also provides a panoramic video free viewing angle display device, wherein the device comprises:

[0030] A scene construction module, used to obtain a geometric projection body constructed for a target real scene in a virtual scene, and to obtain a panoramic video of the target real scene taken by a plurality of cameras;

[0031] A projection module, used for projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection;

[0032] A view area calculation module, used to obtain a user's real-time viewpoint and a user's real-time sight direction, and to perform a correction projection from a camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction;

[0033] The rendering and display module is used to perform scene rendering on the virtual space after the corrected projection and display the rendered picture.

[0034] The present application also provides a terminal, which includes: a memory, a processor, and a panoramic video free-viewing-angle display program stored in the memory and executable on the processor, wherein the panoramic video free-viewing-angle display program implements the steps of the panoramic video free-viewing-angle display method as described above when executed by the processor.

[0035] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the panoramic video free-viewing-angle display method as described above.

[0036] The present invention provides a method, device, terminal and storage medium for displaying panoramic videos in a free viewing angle, the method comprising: obtaining a geometric projection body constructed for a target real scene in a virtual scene, and obtaining a panoramic video of the target real scene shot by several cameras; projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection; obtaining a user's real-time viewpoint and a user's real-time line of sight direction, and completing a corrective projection from the camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time line of sight direction; performing scene rendering on the virtual space after the corrective projection and displaying the rendered picture. The present invention performs a corrective projection of the user's real-time viewpoint in a virtual space with a geometric projection body and a panoramic image projection, and the selectable user viewpoint position is not constrained by the camera layout, thereby enabling the user to move and watch freely in the virtual space, thereby improving the accuracy and continuity of scene space perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a preferred embodiment of the panoramic video free viewing angle display method of the present invention;

[0038] Figure 2 It is the coal mining working face of the underground coal mine and the corresponding geometric projection body in the present invention;

[0039] Figure 3 It is the hemispherical large-scale building and the corresponding geometric projection body in the present invention;

[0040] Figure 4 It is a combination of the corridor and a plurality of classrooms of different sizes and the corresponding geometric projection body in the present invention;

[0041] Figure 5 It is a front view of the large-scale panoramic video virtual scene construction of the underground coal mine in the present invention.

[0042] Figure 6 yes Figure 5 Axis section diagram of the virtual scene in the large-scale panoramic video of the underground coal mine at C1T1.

[0043] Figure 7 It is a schematic diagram of the free movement area of ​​the virtual scene in the present invention.

[0044] Figure 8 It is a functional principle block diagram of a preferred embodiment of the panoramic video free viewing angle display device of the present invention;

[0045] Fig. 9 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] The present invention relates to a method for viewing panoramic videos of large scenes in a virtual reality (VR) scene from a free viewpoint, and is particularly suitable for the application of panoramic images in ultra-large scenes. The method can support viewers to move freely in large scenes, and based on the real-time position and sight direction of users, correct the panoramic video images of large scenes and display them based on hardware rendering such as VR glasses, and belongs to the field of graphics rendering and image display technology in the field of computer vision.

[0048] Computer vision-based monitoring systems already have mature application technologies and a wide range of commercial application scenarios. With the development and application of intelligent technologies, intelligent monitoring systems that integrate image processing technology have been widely used in 24-hour intelligent monitoring in a variety of industrial and commercial scenarios. In such applications, the viewing angle of the monitoring video directly determines the monitoring range, thereby determining the amount of information that can be captured in real time. Traditional monitoring videos are based on monocular pinhole or fisheye cameras, which have the disadvantage of limited viewing angles. In order to meet the needs of wide-viewing angles and all-round monitoring, the current mainstream application has realized 360° panoramic video monitoring based on panoramic camera hardware, which has been widely used in transportation hubs, street monitoring and other scenarios.

[0049] Single-point 360° panoramic video solves the problem of limited viewing angle, but there are still certain limitations on the visible distance, and the visible range is limited to the vicinity of the panoramic camera. Therefore, single-point panoramic video alone still cannot meet the overall monitoring needs of some ultra-large scenes, such as underground coal mining faces, large exhibition halls, large transportation hubs, etc. For such application scenarios, panoramic stitching technology based on ultra-large scenes combines and continuously stitches the images of multiple pinhole or panoramic cameras to form a panoramic visualization of the entire scene, which can significantly improve the overall information integrity of multiple camera information in large scenes and eliminate the sense of separation between multiple cameras.

[0050] On the other hand, mixed reality technology has achieved remarkable development. Through virtual reality technology and hardware, it can provide users with immersive scene experience. It has been widely used in entertainment, education and other industries, bringing users a new experience. Combined with panoramic monitoring technology, it also has great development potential in industrial applications such as remote control and panoramic monitoring. In the application field of underground coal mines, the existing perception, control and decision-making technologies are affected by the complex, harsh and changeable conditions underground. The geological exploration accuracy is not high, the sensor reliability is insufficient, and the control and decision-making intelligence is not high. The complete set of equipment cannot adapt to the complex and changeable environment, and personnel are still required to inspect and operate on site to solve complex and critical problems. Combined with large-scene panoramic stitching technology and mixed reality technology, the underground coal mining site can be directly presented to the user, realizing immersive remote operation and monitoring, and effectively improving the efficiency and safety of underground coal mine operations.

[0051] For this type of technical implementation, how to display the stitched panoramic video of a large scene in a VR glasses (head-mounted display) with a mapping relationship that conforms to the characteristics of human vision is a key technology. The present invention proposes an immersive free viewing method suitable for panoramic videos of large scenes, and is particularly suitable for immersive viewing of panoramic images of large scenes (such as large conference halls, long corridors, coal mining faces, etc.) based on VR glasses (head-mounted displays). The present invention addresses the current problems of information fragmentation and discontinuity of scene information when viewing large scene videos separately, and enables users to complete an immersive viewing experience in a virtual scene, which can improve the integrity and continuity of users' perception of large scene environments and information.

[0052] See also Figure 1 , Figure 1 Flowchart of the panoramic video free viewing angle display method in the present invention. Figure 1 As shown, the panoramic video free viewing angle display method described in the embodiment of the present invention includes:

[0053] Step S100: obtaining a geometric projection body constructed for a target real scene in a virtual scene, and obtaining a panoramic video of the target real scene shot by a plurality of cameras.

[0054] The embodiment of the present application constructs a panoramic virtual scene of a large scene by setting a geometric projection body (or called proxy geometry) that is similar to the scene in the virtual scene, and rendering the correct video image on the surface of the geometric projection body, and cooperates with VR glasses (head-mounted display) to achieve immersive free movement and free viewing of the virtual scene.

[0055] In an embodiment of the present application, the shape of the geometric projection body matches the shape of the target real scene.

[0056] The embodiment of the present application selects a reasonable shape of a geometric projection body according to the characteristics of the monitoring scene and the panoramic stitching algorithm adopted. Specifically, the geometric projection body (Proxy Geometry) plays a role similar to a "projection screen" in the VR virtual scene. In the current mainstream 360° single-viewpoint panoramic imaging VR projection technology, the geometric projection body is generally a sphere or a cylinder, and the viewpoint is set at the geometric center of the geometric projection body (such as: the center of the sphere for a sphere and the midpoint of the cylinder axis for a cylinder). In the present invention, the geometric projection body is selected according to the size and shape of the specific monitoring scene of the application. First, an approximate standard geometric body shape is abstracted from the scene shape to approximate the real scene. For the same scene, there may be more than one projection geometry selection; for scenes with more complex shapes, it can be abstracted as a combination of multiple standard geometries.

[0057] like Figure 2 As shown in the figure, it is a coal mining face in an underground coal mine, which is in the shape of a long tunnel. A long cylinder whose axis coincides with the central axis of the coal mining face tunnel is selected as the geometric projection body of the scene. Figure 3 As shown in the figure, it is a large hemispherical building, so the hemispherical shell is selected as the geometric projection body inside the scene. Figure 4 As shown, it is a combination of a corridor and a plurality of classrooms of different sizes, and a combination of a plurality of cuboids is selected as the geometric projection body of the scene. The present invention is not limited to the above scene.

[0058] The embodiments of the present application can restore the spatial information of the scene to a certain extent by abstracting a large real scene into a geometric body or a combination of geometric bodies and mapping the image pixels in combination with the real-world camera position. Compared with the existing viewing method of watching a single camera separately or watching the panoramic stitched video directly on the screen, it can provide users with more venue space shape information.

[0059] like Figure 1 As shown, the panoramic video free viewing angle display method described in this embodiment also includes:

[0060] Step S200: projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection.

[0061] In an embodiment of the present application, before step S200, the step also includes: determining the resolution of the panoramic video, searching a pre-stored resolution-size correspondence according to the resolution, and obtaining a virtual scene size corresponding to the resolution; and adjusting the scale of the geometric projection body in the virtual scene based on the virtual scene size of the geometric projection body.

[0062] Specifically, the overall size of the geometric projection can be scaled in the virtual scene, and the scaling ratio is determined by the resolution of the panoramic video: the higher the resolution of the panoramic video, the higher the resolution in the virtual scene, and the larger the overall size of the virtual scene; conversely, if the resolution of the panoramic video is small, the overall size of the virtual scene also needs to be reduced to adapt to the low resolution. In addition, the ratio of the internal size should be consistent with the real scene to ensure that the ratio in the final rendered scene is also consistent with the real scene.

[0063] In one embodiment of the present application, step S200 specifically includes:

[0064] Step S210: determining a virtual installation position of a virtual camera corresponding to each camera in the geometric projection body based on actual installation positions of a plurality of cameras in the target real scene;

[0065] Step S220, obtaining camera area images captured by a plurality of cameras in the target real scene according to the panoramic video, and obtaining display images corresponding to each of the virtual cameras based on the virtual installation positions of the virtual cameras and the camera area images;

[0066] Step S230: obtaining a virtual space for panoramic image projection based on the display screens corresponding to the virtual cameras.

[0067] The embodiment of the present application is based on a selected geometric projection body, an actual camera installation position C (x, y, z), and a (stitched) large-scale scene panoramic video (or frame image), and solves the mapping relationship A from panoramic video pixels to geometric projection body position coordinates in units of pixels: P (x, y, z) = A (u, v).

[0068] Specifically, after determining the specific shape and size of the geometric projection body, determine its corresponding position C' in the virtual scene based on the installation position C of the actual camera in the real world. Based on the principle of reproducing the pixels captured by the camera in the corresponding area of ​​the real scene in the virtual scene, determine the coordinate pixel mapping relationship A between the pixels in the panoramic video and the geometric projection body: P(x,y,z)=A(u,v). Figure 2 The following is an example of an underground coal mine scenario.

[0069] like Figure 5 As shown, Figure 5 The projection process of the real scene of an underground coal mine into a VR virtual scene is demonstrated. Figure 5This is a commonly used tunnel camera installation scheme for coal mining working faces (the camera position and number are only used for illustration and will be determined according to actual conditions in actual applications). In the scenario described, the six cameras C1 to C6 are responsible for real-time monitoring of the six cylindrical areas in the tunnel, i.e., the C1 camera shoots the B1B2 cylindrical section, the C2 camera shoots the B2B3 cylindrical end, and so on. Since this scenario uses a cylindrical surface whose axis coincides with the axis of the coal mining face tunnel as the geometric projection body, as shown in Figure 2 As shown. Based on the principle of reproducing the pixels captured by the camera in the corresponding area of ​​the real scene in the virtual scene, the corresponding positions of each camera in the virtual scene are C1'-C6'. The B1'B2' cylindrical segment pixels in the VR virtual scene are provided by the C1 camera screen, the B2'B3' cylindrical segment is provided by the C2 camera, and so on. In this way, by setting the starting position B1' in the virtual scene and determining the corresponding relationship with the real position B1, the display pixels P of all positions (x, y, z) in the scene and the real camera C can be calculated. i The mapping relationship between the picture pixels (u, v) is A: P(x, y, z) = A(u, v, C i ), and then obtain the display screen corresponding to each of the virtual cameras.

[0070] The coal mining face Figure 5 The C1T1 axis section is cut, and the imaging effect is as follows Figure 6 The pixel coverage area on the geometric projection in the virtual scene is determined by the real camera shooting area. In this practical example, the real scene camera shooting area is as follows: Figure 6 As shown in the left figure, the arc segment R can be covered in the cross section 1L T1R 1R , using the method of the embodiment of the present application, the large scene panoramic image projected in the virtual scene can cover an area of ​​arc segment R 1L 'T1'R 1R '. According to the scene requirements, the actual shooting range can be expanded by using a camera with a wider shooting range, that is, the arc segment R can be covered in the cross section 1L T1R 1R ,Using a panoramic 360° camera, the shooting range can be expanded to the entire cross-sectional circle at most, thereby correspondingly expanding the field of view restored in the virtual scene.

[0071] The large-scene panoramic video mentioned in the embodiment of the present application refers to a panoramic video or frame image that has been stitched. The embodiment of the present application is adapted to all ultra-large-scene stitching algorithms that can geometrically project panoramic video images. The specific stitching algorithm is related to the shape characteristics of the actual scene and the camera arrangement method.

[0072] After the embodiment of the present application abstracts the large scene into a geometric projection body, it is possible to project image content onto the surface of the projection body to restore the panoramic content of the large scene.

[0073] like Figure 1 As shown, the panoramic video free viewing angle display method described in this embodiment also includes:

[0074] Step S300, obtaining the user's real-time viewpoint and the user's real-time sight direction, and completing a corrective projection from the camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction.

[0075] In the embodiment of the present application, before step S300, the method further includes: pre-dividing a user viewpoint selection range area in the virtual space.

[0076] The embodiment of the present application can define the area where the user can move freely, that is, the user viewpoint selection range area, in the VR virtual space according to the shape and size of the geometric projection body according to usage requirements.

[0077] Specifically, in the virtual space where the geometric projection surface of the panoramic image projection has been completed, the area where the user's viewpoint can move freely can be delineated according to the specific needs of the application scenario, thereby enhancing the user's immersive experience in the virtual scene.

[0078] like Figure 7 As shown, in the traditional method, the user's viewpoint can only be selected from six camera positions C1' to C6', while using the method of the embodiment of the present application, there are several optional solutions: First, the free movement area is the line segment S0'E0', and any point on the line segment can be selected as the viewing position (such as U1), and the real-time image can be freely viewed (the direction of sight is as shown in FIG. ); Second, the free movement area is the line segment S1'E1', and any point on the line segment can be selected as the viewing position (such as U2), and the real-time image can be viewed freely (the direction of sight is such as ); Third, the free movement area is a cylindrical space area with a diameter of h between S0'E0' and S1'E1'. Any point in the space can be selected as the viewing position, that is, the viewpoint, to freely view the real-time image; Fourth, a subset of the above schemes or other spatial domain combinations can be selected according to needs.

[0079] The above options all allow users to move freely in virtual scenes and watch panoramic videos of large scenes. At the same time, they provide users with the relatively correct spatial positions of objects in the scene, enhance the user's immersive sense of presence, and facilitate a more accurate and comprehensive understanding of the scene space.

[0080] In the embodiment of the present application, the step S300 specifically includes:

[0081] Step S310: obtaining the user's real-time viewpoint and the user's real-time sight direction, determining the real-time viewpoint position of the user's real-time viewpoint in the virtual space, and assuming the real-time viewpoint position as a viewpoint virtual camera;

[0082] Step S320, determining a target virtual camera corresponding to the viewpoint virtual camera in the real-time sight direction of the user, and determining a target camera corresponding to the target virtual camera in a target real scene;

[0083] Step S330: completing the correction projection from the target virtual camera to the viewpoint virtual camera based on the camera parameters of the target camera.

[0084] The embodiment of the present application integrates the user's real-time virtual position U (x, y, z) and the line of sight direction l (α, β, γ) to complete the correction projection. Specifically, based on the principle of visual imaging, according to the user's real-time position and line of sight direction, the homography correction matrix with the camera position and its corresponding picture is solved and the projected picture is corrected to provide a correct perspective relationship so that the user has a correct spatial perception in the virtual scene.

[0085] The embodiment of the present application assumes the real-time viewpoint position as the viewpoint virtual camera, completes the correction projection from the target virtual camera to the viewpoint virtual camera, so that the user can move continuously and freely in the virtual panoramic video scene and freely view the scene in the space without being restricted by the actual installation position of the camera. This solves the problem that in the current large-scale scene panoramic video immersive viewing method, the user cannot move continuously and freely in the virtual VR scene and freely change the direction of sight due to the restriction of the actual installation position of the camera.

[0086] In an embodiment of the present application, the step of completing the corrective projection from the target virtual camera to the viewpoint virtual camera based on the camera parameters of the target camera specifically includes: determining the homography matrix between the pixels photographed by the target virtual camera and the pixels photographed by the viewpoint virtual camera based on the camera parameters of the target camera; and completing the corrective projection from the target virtual camera to the viewpoint virtual camera based on the homography matrix.

[0087] Specifically, Figure 7 As shown, let the user's location be U1 and the sight direction be That is, the viewpoint virtual camera is at U1, and the area viewed in the virtual scene is the cylindrical segment B3'B4', which is captured by the camera at C3'. In order to accurately display the content seen when the viewpoint virtual camera is at U1, the correction is equivalent to correcting the cylindrical segment B3'B4' image taken at C3' to the cylindrical segment B3'B4' image seen by the camera at U1. The homography matrix can be used to complete this process. The homography correction process is as follows:

[0088] For a pixel (u, v) that has been projected onto a geometric projection body, the corrected pixel (u', v') when rendering the image satisfies the homography relationship:

[0089]

[0090] In the formula, Z U , Z C K is the depth value of the pixel to be corrected in the U1 camera coordinate system and the C3 camera coordinate system; U , K C is the camera internal parameter matrix at U1 and C3. In the method of the present invention, K U =K C ; K UC It is the external parameter matrix of point U1 relative to point C3, which contains the camera rotation and translation information. The rotation can be fully described by the three Euler angles of l1 (α, β, γ), and the translation can be fully described by the coordinates of the two cameras in the virtual space.

[0091] The embodiment of the present application sets the free movement area and viewing angle based on the specific scene requirements, and uses the imaging principle to correct the viewport image to restore the accurate perspective relationship, so that the user can watch the panoramic image of the large scene as if they were there, which can greatly enhance the user's immersive sense of presence in the virtual scene, and can accurately judge the positional relationship of various objects and events in space, effectively improving the accuracy of perceived information.

[0092] like Figure 1 As shown, the panoramic video free viewing angle display method described in this embodiment also includes:

[0093] Step S400: performing scene rendering on the virtual space after the corrected projection and displaying the rendered image.

[0094] Specifically, based on three-dimensional scene software such as Unreal Engine (UE), the virtual space where the geometric projection body is located is rendered according to the user's position and line of sight in the virtual scene, that is, the image in the viewport of the user's VR glasses (head-mounted display) is rendered in real time and displayed.

[0095] The embodiments of the present application utilize existing rendering technology and software to enable users to move and view freely in a virtual space where geometric projection bodies are set, provide a strong immersive presence experience, and effectively improve the accuracy and continuity of scene space perception.

[0096] It is understandable that the use of 3D reconstruction technology can also achieve the complete reconstruction of large scenes to form a virtual space, but the 3D reconstruction method has a large amount of calculation and low calculation efficiency, which cannot meet the calculation efficiency requirements of many scenarios. In particular, real-time monitoring application scenarios, such as coal mining face applications, require extremely low latency (within 200ms) or real-time completion of large-scale scene panoramic images to effectively monitor the working face situation, and the 3D reconstruction method cannot achieve the above processing process.

[0097] The embodiment of the present application uses a projection correction algorithm to correct and re-project panoramic videos of large scenes according to changes in the user's position and line of sight, thereby achieving an immersive VR viewing experience with a freely movable perspective.

[0098] In one embodiment, if Figure 8 As shown, based on the above-mentioned panoramic video free viewing angle display method, the present invention also provides a panoramic video free viewing angle display device, including:

[0099] The scene construction module 100 is used to obtain a geometric projection body constructed for a target real scene in a virtual scene, and to obtain a panoramic video of the target real scene taken by a plurality of cameras;

[0100] The projection module 200 is used to project the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection;

[0101] The view area calculation module 300 is used to obtain the user's real-time viewpoint and the user's real-time sight direction, and complete the correction projection from the camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction;

[0102] The rendering and display module 400 is used to perform scene rendering on the virtual space after the corrected projection and display the rendered image.

[0103] Fig. 9 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0104] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0105] When the processor 502 executes the program, the panoramic video free viewing angle display method provided in the above embodiment is implemented.

[0106] Furthermore, the terminal further includes:

[0107] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0108] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0109] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0110] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0111] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0112] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0113] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above-mentioned panoramic video free-viewing-angle display method is implemented.

[0114] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0115] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0116] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0118] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0121] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0122] In summary, the present invention discloses a method, device, terminal and storage medium for displaying panoramic videos in a free viewing angle, the method comprising: obtaining a geometric projection body constructed for a target real scene in a virtual scene, and obtaining a panoramic video of the target real scene shot by several cameras; projecting the panoramic video to the geometric projection body to obtain a virtual space for panoramic image projection; obtaining the user's real-time viewpoint and the user's real-time line of sight direction, and completing a corrective projection from the camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time line of sight direction; performing scene rendering on the virtual space after the corrective projection and displaying the rendered picture. The present invention performs a corrective projection of the user's real-time viewpoint in a virtual space in which a geometric projection body is set and which has a panoramic image projection, and the selectable user viewpoint position is not constrained by the camera layout, thereby enabling the user to move and watch freely in the virtual space, thereby improving the accuracy and continuity of scene space perception.

[0123] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for displaying panoramic video with free viewing angle, characterized in that: The method comprises: Acquire a geometric projection body constructed for a target real scene in a virtual scene, and acquire a panoramic video of the target real scene captured by a plurality of cameras; Projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection; Acquire a user's real-time viewpoint and a user's real-time sight direction, and complete a correction projection from a camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction; The virtual space after the corrected projection is subjected to scene rendering and the rendered image is displayed.

2. The panoramic video free viewing angle display method according to claim 1, characterized in that: The shape of the geometric projection body matches the shape of the target real scene.

3. The panoramic video free viewing angle display method according to claim 1, characterized in that: Before projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection, the method further includes: Determine the resolution of the panoramic video, and search a pre-stored resolution-size correspondence relationship according to the resolution to obtain a virtual scene size corresponding to the resolution; The scale of the geometric projection volume is adjusted in the virtual scene based on the virtual scene size of the geometric projection volume.

4. The panoramic video free viewing angle display method according to claim 1, characterized in that: Projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection includes: Based on actual installation positions of a plurality of cameras in the target real scene, determining a virtual installation position of a virtual camera corresponding to each camera in the geometric projection body; Obtain camera area images captured by a plurality of cameras in the target real scene according to the panoramic video, and obtain display images corresponding to each of the virtual cameras based on the virtual installation positions of the virtual cameras and the camera area images; A virtual space for panoramic image projection is obtained based on the display screens corresponding to the virtual cameras.

5. The panoramic video free viewing angle display method according to claim 4, characterized in that: Acquiring a user's real-time viewpoint and a user's real-time sight direction, and completing a correction projection from a camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction, including: Acquire the user's real-time viewpoint and the user's real-time sight direction, determine the real-time viewpoint position of the user's real-time viewpoint in the virtual space, and assume the real-time viewpoint position as a viewpoint virtual camera; Determine a target virtual camera corresponding to the viewpoint virtual camera in the real-time sight direction of the user, and determine a target camera corresponding to the target virtual camera in a target real scene; A corrective projection from the target virtual camera to the viewpoint virtual camera is completed based on the camera parameters of the target camera.

6. The panoramic video free viewing angle display method according to claim 5, characterized in that: The method further comprises: performing a correction projection from the target virtual camera to the viewpoint virtual camera based on the camera parameters of the target camera, comprising: Determine, based on the camera parameters of the target camera, a homography matrix between pixels photographed by the target virtual camera and pixels photographed by the viewpoint virtual camera; A corrective projection from the target virtual camera to the viewpoint virtual camera is completed based on the homography matrix.

7. The panoramic video free viewing angle display method according to claim 1, characterized in that: The method further comprises: obtaining a user's real-time viewpoint and a user's real-time sight direction, and before completing a correction projection from a camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction, the method further comprises: A user viewpoint selection range area is pre-divided in the virtual space.

8. A panoramic video free viewing angle display device, characterized in that: The device comprises: A scene construction module, used to obtain a geometric projection body constructed for a target real scene in a virtual scene, and to obtain a panoramic video of the target real scene taken by a plurality of cameras; A projection module, used for projecting the panoramic video onto the geometric projection body to obtain a virtual space for panoramic image projection; A view area calculation module, used to obtain a user's real-time viewpoint and a user's real-time sight direction, and to perform a correction projection from a camera of the target real scene to the user's real-time viewpoint in the virtual space based on the user's real-time viewpoint and the user's real-time sight direction; The rendering and display module is used to perform scene rendering on the virtual space after the corrected projection and display the rendered picture.

9. A terminal, characterized in that: include: A memory, a processor, and a panoramic video free-viewing-angle display program stored in the memory and executable on the processor, wherein the panoramic video free-viewing-angle display program, when executed by the processor, implements the steps of the panoramic video free-viewing-angle display method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the panoramic video free-viewing-angle display method according to any one of claims 1 to 7.

Images