Large-scene panoramic stitching method and device, terminal and storage medium

Abstract

The invention provides a large-scene panoramic stitching method and device, a terminal and a storage medium. The large-scene panoramic stitching method comprises the following steps: acquiring panoramic spherical videos acquired by a plurality of panoramic acquisition devices; determining a panoramic spherical picture corresponding to each panoramic spherical video at each frame moment, and unfolding each panoramic spherical picture to obtain a panoramic plane picture; fusing the panoramic plane pictures at each frame moment to obtain a large-scene spliced picture at each frame moment; and synthesizing the large-scene spliced pictures under all frame moments into a large-scene spliced video. According to the invention, all angles in the scene can be monitored by expanding and fusing the panoramic spherical videos acquired by the plurality of panoramic acquisition devices, so that scene information is ensured to be transmitted to the rear end in a high-resolution manner without limiting the visual angle, and all contents in the scene are imaged.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic stitching, and in particular to a large-scene panoramic stitching method, device, terminal and storage medium. Background Art

[0002] With the development of science and technology, panoramic stitching technology is used in all walks of life. Since the viewing angle of a single lens is limited, in order to display more content in one picture, it is necessary to stitch the pictures of multiple cameras. The stitching process involves single picture distortion correction, image matching, image fusion and other technologies.

[0003] Currently, there are three types of panoramic stitching technologies:

[0004] The first is to use a camera to rotate in place and take multiple photos continuously, and then stitch these multiple photos together to form a panoramic picture with a horizontal field of view greater than that of a single camera, such as the panoramic mode of a mobile phone. This panoramic stitching technology is suitable for shooting static scenes and can only generate pictures, not videos.

[0005] The second is to use multiple cameras to form a panoramic lens. The videos of all the lenses are corrected and stitched to form a panoramic picture with a horizontal field of view of 360 degrees and a vertical field of view of less than or equal to 180 degrees. Through real-time correction and stitching, a panoramic video is formed. This video generally needs to be used with a VR headset to achieve a good viewing experience. There will be some distortion when playing it on an ordinary monitor. However, for scenes far away from the device, the resolution is low and it is not suitable for large scene imaging.

[0006] The third type is that ordinary cameras distributed in different positions shoot a flat scene. There are repeated areas between the cameras for image fusion, and finally form a picture or video with a larger horizontal and vertical field of view. The picture or video can be played directly on an ordinary display, and the overall effect is similar to a bird's-eye view. For example, reversing panorama, satellite map, etc. However, when stitching a large plane, such as reversing panorama, only the ground can be stitched, and objects that are far away or at a certain height will produce serious distortion. In addition, for panoramic map technology, the picture is static, and each imaging has a certain distance interval. Therefore, when viewing the scene, the perspective change is discontinuous, and it is impossible to achieve a smooth and continuous change of perspective.

[0007] Therefore, existing panoramic stitching methods cannot ensure that scene information is transmitted to the back end in high resolution without restricting the viewing angle and image all the content in the scene.

[0008] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0009] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a large-scene panoramic stitching method, device, terminal and storage medium are provided, aiming to solve the problem that the large-scene panoramic stitching method in the prior art cannot ensure that the scene information is transmitted to the back end in high resolution without limiting the viewing angle, and image all the content in the scene.

[0010] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0011] A large scene panoramic stitching method, wherein the method comprises:

[0012] Obtain panoramic spherical videos collected by several panoramic collection devices;

[0013] Determine the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time, and expand each of the panoramic spherical pictures to obtain a panoramic plane picture;

[0014] The panoramic plane images at each frame time are fused to obtain a large scene splicing image at each frame time;

[0015] The large scene stitching pictures at all frame times are combined into a large scene stitching video.

[0016] In one embodiment of the present application, a plurality of the panoramic acquisition devices are arranged in parallel in sequence, installed horizontally and spaced apart by a preset distance.

[0017] In one embodiment of the present application, determining the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time includes:

[0018] The panoramic spherical videos are time-synchronized to obtain the panoramic spherical images corresponding to the panoramic spherical videos at each frame time.

[0019] In one embodiment of the present application, the panoramic plane images at each frame time are fused to obtain a large scene spliced ​​image at each frame time, including:

[0020] Calculate the homography matrix between adjacent panoramic plane images at the same viewing angle at each frame moment;

[0021] According to each of the homography matrices, two adjacent panoramic plane images are fused in sequence to obtain a large scene spliced ​​image at each frame time.

[0022] In one embodiment of the present application, calculating the homography matrix between adjacent panoramic plane images at the same viewing angle at each frame moment includes:

[0023] Using a feature extraction algorithm to extract feature points of two adjacent panoramic plane images at each frame time;

[0024] Matching feature points of two adjacent panoramic plane images to obtain a matching point pair;

[0025] A homography matrix between two adjacent panoramic plane pictures at the same viewing angle at each frame moment is obtained according to the matching point pairs.

[0026] In one embodiment of the present application, two adjacent panoramic plane images are sequentially fused according to each homography matrix to obtain a large scene spliced ​​image at each frame time, including:

[0027] Performing linear transformation on two adjacent panoramic plane images in sequence according to each homography matrix;

[0028] The linearly transformed panoramic plane images are fused in sequence until all the panoramic plane images at each frame time are fused to obtain a large scene splicing image at each frame time.

[0029] In one embodiment of the present application, after obtaining a large scene stitching video according to the large scene stitching pictures at all frame times, the method further includes:

[0030] Determine a current viewing angle corresponding to a VR device used to display the large-scene spliced ​​video;

[0031] Performing panoramic rendering on the large scene stitching picture in the large scene stitching video in real time according to the current viewing angle to obtain a panoramic rendering video;

[0032] The panoramic rendering video is transmitted to the VR device for display.

[0033] The present application also provides a large scene panoramic stitching device, wherein the device comprises:

[0034] An acquisition module, used to acquire panoramic spherical videos acquired by a plurality of panoramic acquisition devices;

[0035] A determination module, used to determine the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time, and to expand each of the panoramic spherical pictures to obtain a panoramic plane picture;

[0036] A fusion module is used to fuse the panoramic plane images at each frame time to obtain a large scene splicing image at each frame time;

[0037] The synthesis module is used to synthesize the large scene stitching pictures at all frame times into a large scene stitching video.

[0038] The present application also provides a terminal, which includes: a memory, a processor, and a large scene panoramic stitching program stored in the memory and executable on the processor, wherein the large scene panoramic stitching program implements the steps of the large scene panoramic stitching method described above when executed by the processor.

[0039] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the large scene panoramic stitching method as described above.

[0040] The present invention provides a large-scene panoramic stitching method, device, terminal and storage medium, the large-scene panoramic stitching method comprising: obtaining panoramic spherical videos collected by a plurality of panoramic acquisition devices; determining the panoramic spherical images corresponding to each of the panoramic spherical videos at each frame time, and expanding each of the panoramic spherical images to obtain a panoramic plane image; fusing the panoramic plane images at each frame time to obtain a large-scene stitching image at each frame time; and synthesizing the large-scene stitching images at all frame times into a large-scene stitching video. The present invention can monitor all angles in the scene by expanding and fusing the panoramic spherical videos collected by a plurality of panoramic acquisition devices, and ensures that the scene information is transmitted to the back end in a high-resolution form without limiting the viewing angle, and images all the contents in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of a preferred embodiment of the large scene panoramic stitching method of the present invention;

[0042] Figure 2 It is a schematic diagram of the installation of several panoramic acquisition devices in the present invention;

[0043] Figure 3 It is a schematic diagram of a panoramic plane picture obtained by unfolding a plurality of panoramic spherical pictures in the present invention;

[0044] Figure 4 It is a schematic diagram of a large scene splicing picture obtained by fusing several panoramic plane pictures in the present invention;

[0045] Figure 5 It is a functional principle block diagram of a preferred embodiment of a large scene panoramic stitching device in the present invention;

[0046] Figure 6 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The safety issue of the coal mining industry has always been a difficult problem that has plagued the coal mining industry. If the abnormal conditions of the coal mining face can be discovered in time, the probability of accidents can be effectively reduced. In order to detect abnormal conditions in time, there are usually two ways: 1. The staff moves with the coal mining machine and observes the conditions of the various parts before and after the coal mining machine at any time, but this method introduces another personnel risk; 2. Install cameras at appropriate locations to transmit real-time image data to a safe back-end. The back-end personnel observe the conditions of the various parts before and after the coal mining machine through the monitor, so that the staff does not need to follow the coal mining machine all the time, thereby reducing the risk.

[0049] The monitoring screen is actually a two-dimensional projection of the real three-dimensional scene. The video screen is transmitted back to the back-end monitor for display. This display method is very different from the viewing experience of people in the actual scene, especially when there are many cameras, which brings a lot of workload to the monitoring personnel. If the images of multiple cameras can be spliced ​​and then displayed, it can not only reduce the bandwidth during network transmission, but also improve the efficiency of monitoring personnel.

[0050] The present invention is mainly aimed at large scenes such as coal mining working faces. Such scenes are generally long and narrow passages (generally tens to hundreds of meters), and it is usually impossible to use a single lens to obtain high-resolution scene videos of the entire working face. Some panoramic devices can capture 360-degree panoramic images in the scene, but due to their limited resolution, scenes far away from the panoramic device cannot be clearly imaged.

[0051] Therefore, the present invention sets a certain interval and sets up multiple panoramic devices to obtain multiple panoramic images at the same time, and then splices them into a large scene panorama. In this way, all scenes of the coal mining face can be transmitted to the back end in real time, and all parts of the scene can be clearly imaged.

[0052] See also Figure 1 , Figure 1 Flowchart of the large scene panoramic stitching method in the present invention. Figure 1 As shown, the large scene panoramic stitching method described in the embodiment of the present invention includes:

[0053] Step S100: Obtain panoramic spherical videos captured by a plurality of panoramic capture devices.

[0054] In the embodiment of the present application, several panoramic acquisition devices are arranged in parallel, installed horizontally and spaced by a preset distance. In this way, for a narrow and long coal mining face, multiple panoramic lenses can be set up on the entire coal mining face to obtain high-definition videos of various parts of the scene, so as to facilitate panoramic stitching of the narrow and long scene and generate real-time, high-resolution panoramic images.

[0055] When installing panoramic data collection equipment on coal mining surfaces, Figure 2 As shown, a bracket is provided on the top of the coal mining face for installing a panoramic acquisition device 1. In the embodiment of the present application, multiple panoramic acquisition devices are set up at a certain distance to collect high-definition videos of various parts of the scene. Since the images at the two poles are most severely distorted in the spherical panorama, in order to splice the images with almost no distortion near the equator, the panoramic acquisition device is installed horizontally.

[0056] The embodiment of the present application arranges several panoramic acquisition devices in parallel in sequence, so that when acquiring video of a long and narrow scene of a coal mining face, it is possible to obtain the acquired video of each part, and then the entire video is summed to obtain a complete coal mining face monitoring video.

[0057] like Figure 1 As shown, the large scene panoramic stitching method described in this embodiment also includes:

[0058] Step S200: determining the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time, and unfolding each of the panoramic spherical pictures to obtain a panoramic plane picture.

[0059] Specifically, in order to freely change the viewing angle when watching, rather than jumping between camera panoramas, the embodiment of the present application expands all panoramic videos according to the expansion model, such as Figure 3 As shown, an equal number of rectangular panoramic videos are obtained, and a large-scene panoramic video is formed by splicing and fusing these rectangular panoramic videos.

[0060] In an embodiment of the present application, determining the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time includes: time synchronizing each of the panoramic spherical videos to obtain the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time.

[0061] Specifically, since large scenes are stitched in a dynamic environment, time synchronization is required between multiple devices to prevent a target from appearing twice in the picture. Time synchronization mainly includes software synchronization and hardware synchronization. For software synchronization, it can be implemented in a queue manner. By setting a queue for each panoramic video stream, each panoramic decoded video frame is placed in the corresponding queue. When stitching, the latest frames in each queue are spliced. In theory, the maximum time error of all frames obtained in this way is the interval length of two frames. Therefore, the higher the frame rate, the more accurate the time synchronization, but it also brings negative effects of large data volume and large calculation amount. The general panoramic lens acquisition frame rate is 24fps or 30fps. For hardware synchronization, it can be implemented through the Global Navigation Satellite System (GNSS), which provides accurate time information by receiving satellite signals. This method requires the panoramic acquisition device to have a built-in GNSS receiving module. This method can ensure the time consistency between multiple panoramic devices.

[0062] like Figure 1 As shown, the large scene panoramic stitching method described in this embodiment also includes:

[0063] Step S300: Fusing the panoramic plane images at each frame time to obtain a large scene spliced ​​image at each frame time.

[0064] In the embodiment of the present application, the step S300 specifically includes:

[0065] Step S310, calculating the homography matrix between adjacent panoramic plane pictures at the same viewing angle at each frame moment;

[0066] Step S320: fuse two adjacent panoramic plane images in sequence according to each homography matrix to obtain a large scene spliced ​​image at each frame time.

[0067] Specifically, after the image is transformed by the homography matrix, what is obtained is the projection image of the original image at another perspective, or an image that has undergone perspective transformation. The homography matrix is ​​a 3×3 matrix that describes the mapping relationship from one plane to another. It is usually used in scenes such as image correction, image stitching, and image positioning in augmented reality (AR). In addition, it is used to rotate, scale, translate, and project and distort an image. The transformation formula from the original pixel coordinate (x, y) to the transformed pixel coordinate (x′, y′) is as follows:

[0068]

[0069] The homography matrix H of the embodiment of the present application can describe the transformation relationship between the two image planes, including rotation, scaling and translation, etc. By accurately calculating the homography matrix, two or more images can be aligned at the pixel level to achieve seamless splicing; after transformation using the homography matrix, the visual difference at the seam can be minimized, making the spliced ​​image more natural and smooth.

[0070] In one embodiment of the present application, step S310 specifically includes:

[0071] Step S311, using a feature extraction algorithm to extract feature points of two adjacent panoramic plane images at each frame time;

[0072] Step S312, matching the feature points of two adjacent panoramic plane images to obtain a matching point pair;

[0073] Step S313: obtaining a homography matrix between two adjacent panoramic plane images at the same viewing angle at each frame moment according to the matching point pairs.

[0074] Specifically, feature points of two adjacent unfolded panoramic plane images are extracted using a feature extraction algorithm, such as SIFT, SURF, ORB, etc., and then the extracted feature points are matched to find the corresponding relationship between adjacent images.

[0075] The embodiment of the present application utilizes a feature point matching algorithm that is generally robust to image rotation, scaling, lighting changes, etc., so the calculated homography matrix is ​​more reliable. Feature point matching does not depend on the specific content or structure of the image. As long as there are enough identifiable feature points in the image, calculations can be performed, which makes the calculation of the homography matrix more general and flexible.

[0076] In the embodiment of the present application, the step S320 specifically includes:

[0077] Step S321, performing linear transformation on two adjacent panoramic plane images in sequence according to each homography matrix;

[0078] Step S322: The panoramic plane images after the linear transformation are sequentially fused until all the panoramic plane images at each frame time are fused to obtain a large scene spliced ​​image at each frame time.

[0079] Specifically, when an image is transformed using a homography, each point in the image is linearly transformed according to the homography, thereby obtaining the position of the corresponding point in the transformed image. Since the homography contains information about rotation, scaling, translation, and perspective transformations, the transformed image may have a different perspective, size, shape, or position than the original image.

[0080] The embodiment of the present application fuses all transformed images one by one, thereby eliminating redundancy and forming a large scene panoramic picture, such as Figure 4 Specifically, first, image 1 is transformed according to the homography matrix H1, and then image 0 and the transformed image 1 are fused to generate fused image 0; then image 2 is transformed according to the homography matrix H2, and then fused image 0 and the transformed image 2 are fused to generate fused image 1; and so on.

[0081] The embodiment of the present application obtains a wider and more comprehensive field of view by stitching images taken at multiple angles, thereby obtaining a complete panoramic video of the coal mining face.

[0082] like Figure 1 As shown, the large scene panoramic stitching method described in this embodiment also includes:

[0083] Step S400: synthesize the large scene stitching images at all frame times into a large scene stitching video.

[0084] In the embodiment of the present application, after step S400, the following steps are further included:

[0085] Determine a current viewing angle corresponding to a VR device used to display the large-scene spliced ​​video;

[0086] Performing panoramic rendering on the large scene stitching picture in the large scene stitching video in real time according to the current viewing angle to obtain a panoramic rendering video;

[0087] The panoramic rendering video is transmitted to the VR device for display.

[0088] Specifically, based on the generated large scene panoramic video, such as Figure 4 As shown, when the viewing angle of the VR device changes (randomly slide on the thick line), the panoramic rendering can be performed based on the point in the panorama in real time, and the panorama can be transmitted to the VR headset. The embodiment of the present application can render to the back end in real time according to the change of the viewing angle. The viewing angle can be freely changed, which is convenient for patrolling the coal mining face and timely discovering the abnormality of the coal mining face.

[0089] Therefore, the method of the embodiment of the present application can not only ensure that the scene information is transmitted to the back end in high resolution without limiting the viewing angle, but also image all the content in the scene, thereby reducing the risk to personnel at the coal mining face and improving the efficiency of detecting abnormalities at the coal mining face, thereby helping to improve safety during coal mining.

[0090] In one embodiment, if Figure 5 As shown, based on the above large scene panoramic stitching method, the present invention also provides a large scene panoramic stitching device, including:

[0091] An acquisition module 100 is used to acquire panoramic spherical videos acquired by a plurality of panoramic acquisition devices;

[0092] The determination module 200 is used to determine the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time, and expand each of the panoramic spherical pictures to obtain a panoramic plane picture;

[0093] A fusion module 300 is used to fuse the panoramic plane images at each frame time to obtain a large scene spliced ​​image at each frame time;

[0094] The synthesis module 400 is used to synthesize the large scene stitching pictures at all frame times into a large scene stitching video.

[0095] Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0096] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0097] When the processor 502 executes the program, the large scene panoramic stitching method provided in the above embodiment is implemented.

[0098] Furthermore, the terminal further includes:

[0099] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0100] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0101] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0103] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0104] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0105] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the large scene panoramic stitching method as described above is implemented.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0108] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0109] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0110] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0113] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0114] In summary, the present invention discloses a large-scene panoramic stitching method, device, terminal and storage medium, and the large-scene panoramic stitching method includes: obtaining panoramic spherical videos collected by a plurality of panoramic acquisition devices; determining the panoramic spherical images corresponding to each of the panoramic spherical videos at each frame time, and expanding each of the panoramic spherical images to obtain a panoramic plane image; fusing the panoramic plane images at each frame time to obtain a large-scene stitching image at each frame time; and synthesizing the large-scene stitching images at all frame times into a large-scene stitching video. The present invention can monitor all angles in the scene by expanding and fusing the panoramic spherical videos collected by a plurality of panoramic acquisition devices, thereby ensuring that the scene information is transmitted to the back end in a high-resolution form without limiting the viewing angle, and imaging all the content in the scene.

[0115] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A large scene panoramic stitching method, characterized in that: The method comprises: Obtain panoramic spherical videos collected by several panoramic collection devices; Determine the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time, and expand each of the panoramic spherical pictures to obtain a panoramic plane picture; The panoramic plane images at each frame time are fused to obtain a large scene splicing image at each frame time; The large scene stitching pictures at all frame times are combined into a large scene stitching video.

2. The large scene panoramic stitching method according to claim 1, characterized in that: The plurality of panoramic acquisition devices are arranged in parallel in sequence, installed horizontally and spaced apart by a preset distance.

3. The large scene panoramic stitching method according to claim 1, characterized in that: Determining the panoramic spherical images corresponding to each of the panoramic spherical videos at each frame time includes: The panoramic spherical videos are time-synchronized to obtain the panoramic spherical images corresponding to the panoramic spherical videos at each frame time.

4. The large scene panorama stitching method according to claim 1, characterized in that: The panoramic plane images at each frame time are fused to obtain the large scene stitching images at each frame time, including: Calculate the homography matrix between adjacent panoramic plane images at the same viewing angle at each frame moment; According to each of the homography matrices, two adjacent panoramic plane images are fused in sequence to obtain a large scene spliced ​​image at each frame time.

5. The large scene panorama stitching method according to claim 4, characterized in that: Calculate the homography matrix between adjacent panoramic plane images at the same viewing angle at each frame moment, including: Using a feature extraction algorithm to extract feature points of two adjacent panoramic plane images at each frame time; Matching feature points of two adjacent panoramic plane images to obtain a matching point pair; A homography matrix between two adjacent panoramic plane pictures at the same viewing angle at each frame moment is obtained according to the matching point pairs.

6. The large scene panorama stitching method according to claim 4, characterized in that: According to each of the homography matrices, two adjacent panoramic plane images are sequentially merged to obtain a large scene splicing image at each frame time, including: Performing linear transformation on two adjacent panoramic plane images in sequence according to each homography matrix; The linearly transformed panoramic plane images are fused in sequence until all the panoramic plane images at each frame time are fused to obtain a large scene spliced ​​image at each frame time.

7. The large scene panorama stitching method according to claim 1, characterized in that: After obtaining a large scene splicing video according to the large scene splicing pictures at all frame times, the method further includes: Determine a current viewing angle corresponding to a VR device used to display the large-scene spliced ​​video; Performing panoramic rendering on the large scene stitching picture in the large scene stitching video in real time according to the current viewing angle to obtain a panoramic rendering video; The panoramic rendering video is transmitted to the VR device for display.

8. A large scene panoramic stitching device, characterized in that: The device comprises: An acquisition module, used to acquire panoramic spherical videos acquired by a plurality of panoramic acquisition devices; A determination module, used to determine the panoramic spherical picture corresponding to each of the panoramic spherical videos at each frame time, and to expand each of the panoramic spherical pictures to obtain a panoramic plane picture; A fusion module is used to fuse the panoramic plane images at each frame time to obtain a large scene splicing image at each frame time; The synthesis module is used to synthesize the large scene stitching pictures at all frame times into a large scene stitching video.

9. A terminal, characterized in that: include: A memory, a processor, and a large scene panoramic stitching program stored in the memory and executable on the processor, wherein the large scene panoramic stitching program, when executed by the processor, implements the steps of the large scene panoramic stitching method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the large-scene panoramic stitching method as described in any one of claims 1 to 7.

Images