Generating a panoramic image using a multi-faceted sensor system

Abstract

A method for generating a panoramic image includes capturing a video signal using at least two multi-faceted modules, transmitting the unprocessed video signal to a computing unit, processing the video signal, and compositing all video signals into a single video stream in an equirectangular projection format to obtain a composite video signal, wherein each video signal is subject to processing for the correction and compensation of distortions, the transformation of coordinate systems into projections with bilinear interpolation, gradient blending of colour channels at crossover points, and image filtering according to the functions of colour palette, brightness, saturation and contrast. The technical result is an increase in the spatial and energy resolution of the panoramic image, and the transmission of the image with minimal delay that is as near as possible to real-time.

Description

[0001] FORMATION OF A PANORAMIC IMAGE USING A MULTIFACET SENSOR SYSTEM

[0002] The group of inventions relates to the processing of digital images, and more specifically to methods for forming panoramic images from a video stream of frames obtained using a multi-faceted sensor system for the subsequent composition of the obtained images into a single coded signal in an equidistant projection format.

[0003] Panoramic systems with a discrete (facet) angular field consist of numerous optical channels, each of which perceives the radiation flux in a relatively narrow angular field. The optical axes of the channels are rotated relative to each other at a certain angle and together cover a wide angular field.

[0004] The development of faceted optical-electronic multichannel systems with focal matrices is rapidly developing. Such systems use macrooptical lenses that create an image on a photodetector matrix. The overall panoramic image in this system is created by parts that are combined into a general picture using digital processing in real time. Macrooptical systems allow obtaining a panoramic image with a much higher spatial and energy resolution than microoptical ones. Such systems can be used in threat detection systems, monitoring, target designation, for panoramic photo and video shooting, cavity control, etc. Prospects in the development of systems with focal matrices are associated with the creation of large-format photodetector matrices and the development of computing capabilities of computer technologies.

[0005] The following terms and definitions are used in the application materials.

[0006] Image blending is a method that allows you to insert part of one image into another in such a way that the composition of the images looks natural, without seams at the boundaries of the insert.

[0007] Distortion (from Latin distorsio, distortio — curvature) is an aberration of optical systems, in which the coefficient of linear magnification changes as the displayed objects move away from the optical axis. Pipeline image processing is a technology for accelerated processing of multi-tone images, in which the image is considered as two-level in the first step, four-level in the second, etc. [L.M. Nevdyaev. Telecommunication technologies. English-Russian explanatory dictionary and reference book. Edited by Yu.M. Gornostaev. Moscow, 2002].

[0008] CPU (central processing unit) - central processor (CPU; also central processing unit - CPU).

[0009] GPU (graphics processing unit) - graphic processor.

[0010] BAYER - Bayer filter (Bayer pattern) is a two-dimensional array of color filters that cover the photodiodes of photo matrices. It is used to obtain color images in matrices of digital cameras, video cameras and scanners.

[0011] SIMD (single instruction, multiple data) is a principle of computer computing that allows for parallelism at the data level. One of the classes of computing systems in Flynn's classification.

[0012] The closest analogue to the claimed method is the technical solution known from patent RU 2442023, consisting in forming panoramic images from a video stream of frames in real time. Sets of characteristic elements are identified on the frame images, the image of the current frame is compared with the resulting image, the coordinates of the displacement vector of the current frame are determined and the image of the current frame is shifted by the calculated coordinates of the displacement vector, the boundary of the overlap area is calculated and the shifted image of the current frame is added to the resulting one. Characteristic rectangular areas are constructed around the identified characteristic elements, sets of characteristic rectangular areas are compared, while the image of the current frame is compared only with neighboring reference frames and only in the areas of interest of the images of the current and reference frames, the value of the average intensity of the pixels of the image of the current frame is calculated.The coordinates of the current frame displacement vector relative to the adjacent reference frames are determined. The image of the current frame is shifted relative to the images of the adjacent reference frames and added to the resulting image.

[0013] However, this solution is not intended for multifaceted systems.

[0014] The closest analogue to the claimed system is the technical solution known from patent US 6693273 Each of the plurality of individually selectable emitters is adapted to provide a narrow beam, the angle of which relative to the horizontal plane is shifted relative to the other emitters. All emitted beams lie essentially in the same plane in azimuth. The controller responds to certain stimuli to selectively activate one or more emitters. The energy generated in this way is controlled by a receiver, preferably located in the same housing. The output signal of the receiver is analyzed by the controller to determine the presence of an obstacle in that part of the aperture that is illuminated by the selected emitter(s). A plurality of systems can provide an input signal to the controller in order to influence which of the plurality of emitters should be activated to detect an object.

[0015] However, this solution is not intended to form a panoramic image.

[0016] The technical problem of the claimed invention is the creation of a technical solution that ensures the capture, processing and encoding of panoramic images with a coverage of 360 degrees and the formation of a video signal suitable for transmission over the Internet for further playback in the format of panoramic video of 360 degrees and use in virtual reality, which is ensured by turning the panoramic image of the video of 360 degrees from equidistant to volumetric projection.

[0017] The technical result of the group of inventions is an increase in the spatial and energy resolution of a panoramic image during its transmission with the lowest possible delay, close to real-time mode.

[0018] The claimed technical result is achieved due to the fact that the method for forming a panoramic image includes: processing a high-resolution video signal using at least two multifaceted modules, transmitting the unprocessed video signal to the computing unit, processing the video signal, which uses calculations with a single command stream, a multiple data stream; composing all video signals into a single video stream in the equidistant projection format, obtaining a composite video signal, wherein each video signal is subjected to serial processing with its own unique parameters for each function: correction and compensation of distortions in accordance with the technical features of the optics of the multifaceted module; transforming coordinate systems into projections with bilinear interpolation; gradient mixing of color channels at intersections; filtering the image by functions: color palette, brightness and saturation, clarity.

[0019] The following additions can also be used in the claimed method.

[0020] Captures audio signal and synchronizes it with composite video signal.

[0021] They implement signal encoding by transmitting uncompressed audio and video signals to the selected encoder.

[0022] They broadcast audio and video signals to the Internet.

[0023] The claimed technical result is also achieved due to the fact that the system for implementing the method contains a multifaceted unit and a computing unit connected by connectors, the multifaceted unit contains at least two multifaceted modules, each of which contains a lens and a video image capture sensor, and each multifaceted module is calibrated and has an adjusted lens.

[0024] The claimed system is capable of transmitting panoramic ultra-high-definition images with full spatial resolution as an encoded signal over the network.

[0025] Internet with the lowest possible latency, close to real time.

[0026] To solve the problem, a new principle of pipeline image processing was developed, allowing the image to be processed and transmitted without loss in quality and time.

[0027] The developed unique software solution for organizing output signals allows for seamless integration of the claimed invention into video surveillance systems, automatic broadcast systems in security and entertainment applications for creating online broadcasts, and real-time videoconferencing systems.

[0028] The algorithm is based on basic stereometric methods optimized for operation in a mode close to real time, i.e. with minimal execution delays.

[0029] A unique property of the algorithm is its universality and the possibility of application for various configurations of facet systems. The developed system for setting up the operation of algorithms allows interactively (during the operation of the algorithms) changing the values ​​of all spatial variables, interpolation depths and spatial segment blending widths. This technology allows manual or automatic adaptation of the performance of algorithms to various hardware solutions for streaming SIMD extensions of processors, optimizations and synchronization of parallel computing on various graphics accelerators and central processors, taking into account the architecture of pipeline processing, providing calculations with a single command stream, multiple data streams.

[0030] The result of the algorithms is an image with full spatial resolution in 32 / 64-bit format with an alpha channel. The maximum image size and frame rate are determined by the computing capabilities of the hardware solutions; the resolution can reach any values ​​in pixels in width and height with a frequency of 25 fps. The resolution and frame rate are limited only by the parameters of the hardware computing platform.

[0031] The essence of the invention is explained by figures 1-7, which show:

[0032] Fig. 1 - an example of pixel values ​​in a signal from one channel depending on the wavelength received from the focal matrix in a multifaceted block;

[0033] Fig. 2 - illustration of a section of pixels in the "raw" BAYER format;

[0034] Fig. 3 - algorithm for forming a panoramic image;

[0035] Fig. 4 - diagram of the formation of an equidistant projection;

[0036] Fig. 5 - image of a multifaceted block;

[0037] Fig. 6-7 - image of industrial modules.

[0038] The system contains a multifaceted unit, connectors in the form of interface wires (cables) and a computing unit.

[0039] Multifaceted unit withcoossttooiitt from several industrial multifaceted modules, each of which contains a lens and a video capture sensor. Each module is calibrated, in a single housing and has adjusted unique optics - a lens, the distinctive feature of which is the high accuracy of the conjugation of the optical characteristics of the optics itself and the photoreceiving element (video capture sensor) taking into account the angle of spatial coverage and the resolution of the optical elements inside the lens.

[0040] The body consists of aluminum parts and fastening elements. The development of our own multifaceted unit is based on the most accurate calculation and adjustment of the optical part for segmented capture of the entire spatial resolution. With the aim of obtaining the maximum quality of the resulting image, photoreceiving matrices from SONY were selected and calculated as part of industrial modules and a working area size of more than one inch.

[0041] The criteria for selecting photodetector matrices were the following indicators:

[0042] - experience and distribution of the company's products in the digital sensor market (Sony is the largest manufacturer of sensors for digital cameras);

[0043] - sensor size (the bigger the better);

[0044] - frame rate that allows the industrial module to be transmitted (the higher, the better);

[0045] - effective resolution (the higher the better);

[0046] - pixel size (the bigger the better).

[0047] Interface cables connect the multifaceted unit to the computing unit via an interface corresponding to the industrial multifaceted modules in the system. In accordance with the technical requirements of each interface, cables may have limitations on their length and installation method.

[0048] The computing unit is similar to the composition of a personal computer, except for the graphic accelerators and the number of processor cores, the manufacturer and technical characteristics of which are determined depending on the composition of the multifaceted unit and the requirements for the output video stream. Contains hardware accelerators.

[0049] The control interface has a network implementation for access to the system settings and commands by a remote operator via an Internet connection. This interface allows you to implement your own server solution for organizing access to the system controls via a personal account on the site.

[0050] The method is carried out as follows.

[0051] The method is implemented using the above-described system and software.

[0052] The software operates on the principle of conveyor production of video and audio content. At the input of the conveyor are separate images in the form of unique fragments of the entire angular field, and at the output is a compressed audio and video signal for playback in the virtual reality format.

[0053] Capture spatial resolution by photodetector matrices in the developed multifaceted block, for example, more than 2 "raw" (uncompressed) video signals from the multifaceted block with an image resolution of 2K (1920 * 1080 and higher with a frequency of at least 25 frames per second). A unique software solution for image capture is capable of working with the multifaceted block via various interfaces, including the most common USB 3.0 and Ethernet. The uniqueness of the solution is achieved through the use of a low-level programming language, due to which it achieves minimal data processing time for working in real time.The program also has the ability to encode the final image in various Internet protocols (RTSP, RTCP, UDP, RTMP, RTMPS, NDI, WebRTC, HLS, HLS low latency), and has the ability to refine it to compatibility with new protocols that have not yet been implemented, distribute signals to several consumers (for example: simultaneous recording processes to a file, streaming to the Internet, the output of the system virtual camera for import into third-party applications), individually configure a large number of parameters of each of the sensors included in the multifaceted block (for example: exposure / shutter speed, light sensitivity, shutter, color temperature, color palette shift, contrast, gamma, resolution, frame rate FPS, light gain, sharpness, black / light / midtone levels), has no restrictions on the degree of compression of the material.

[0054] (the video bitrate during compression is theoretically unlimited, in practice the speed limiting factor and, accordingly, the quality of the video is the speed of the RAM (random access memory) and ROM (read-only memory) of the specific computer on which our software is installed), works with "raw data" (RAW, for example, when using the USB 3.0 interface and two sensors / cameras, we have two streams of 5 Gbit / s each), allows you to connect developed or external plugins for detecting people and cars or any other scenarios for video analytics based on artificial intelligence, implement a two-way connection in video conference mode in virtual reality due to 360-degree video with a resolution of at least 4K via a web interface with minimal delay in real time.

[0055] Transfer data from the multifaceted block to the computing block. The main feature of this stage is the transfer of uncompressed data via high-speed interfaces. This principle allows eliminating quality losses during intermediate compression. The multichannel system transfers primary data from photodetector matrices without any processing.

[0056] Process "raw" data received from each channel of the multifaceted block. Based on hardware accelerators, format and process video signals in accordance with the technical requirements of each sensor in the multifaceted block. At this stage, in order to optimize all image processing algorithms, each video signal is formatted according to the principle of color format translation with decomposition of color components according to its own decomposition template of all color carriers, which provides the optimal number and order of operations with RAM.

[0057] The developed algorithms allow to split calculations into functional blocks. Distribution into parallel calculations in functional blocks is necessary to achieve high speed of processing of large data array. The interpolation model determines the number of computational iterations, and in accordance with the frequency and number of possible blocks, the algorithm is able to determine the optimal interpolation model from all options in order to optimize the work for certain computing modules. The principle of signal processing on hardware accelerators is built taking into account universal compatibility, which makes the entire processing system maximally independent from the hardware manufacturer.

[0058] All video signals are composed into a single video stream in equidistant projection format (2:1 format). In this case, each video signal is subjected to serial processing with its own unique parameters for each function:

[0059] - correction and compensation of all distortions in accordance with the technical features of the optics of the multifaceted block module;

[0060] - transformation of coordinate systems into projections with bilinear interpolation;

[0061] - gradient mixing of color channels at the intersections of adjacent angular sections of a spherical surface;

[0062] - image filtering by functions: color palette, brightness and saturation, clarity.

[0063] Captures audio signal from an external device and synchronizes it with a composite video signal. Output video and audio signals are formatted to meet the requirements of the selected video and audio codec.

[0064] Organize an uninterrupted process of encoding and broadcasting audio and video signals to the Internet using a pre-specified protocol. With the aim of universal application of the invention, an interface for controlling video and sound encoders was developed for software and hardware implementation. At this stage of the pipeline, a mechanism for transmitting uncompressed audio and video signals to a selected encoder with minimal delay in real time was developed. Minimal delay is ensured by multithreading, optimization and high speed of all algorithms for fast stitching of images from several sensors into an equidistant projection of 2:1 format. Within the framework of support for different protocols, a universal interface with the ability to fine-tune their operation was developed. The universality of the interface is understood as the ability to fine-tune image parameters universally for each protocol.

[0065] Supported protocols: RTMP, UDP, SRT, NDI.

[0066] Supported codecs: H.264, H.265.

[0067] Support for hardware encoders: Intel QuickSync (QSV), NVIDIA Codec.

[0068] Audio codec: AAC.

[0069] Maximum number of audio channels: 4.

[0070] The entire audio and video signal production pipeline operates in a mode close to real time, while having a slight delay of 100 ms.

[0071] Fig. 1 shows an example of pixel values ​​in a signal from one channel depending on the wavelength received from the focal matrix in a multifaceted block. Fig. 2 shows an illustration of a section of pixels in the "raw" BAYER format.

[0072] The universality of the algorithm, achieved by introducing into the system the specification parameters required for precise definitions of the execution conditions and the sequence of calculations in the algorithms, allows adapting calculations with the required operating speed to various hardware solutions in the form of a central processor "SRP" and / or a graphic accelerator "GPU".

[0073] An example of an implemented system.

[0074] The multifaceted unit includes industrial modules with 4 / 3-inch sensors / matrices with two aligned macro-optical lenses. The multifaceted unit is configured for two channels with a spatial capture of 180 degrees each. The multifaceted unit is connected to the computing unit via a pair of high-speed USB3.1 interfaces.

[0075] The computing unit with a central processor of class x86 Intel Core i7 / i9 and a graphics accelerator of class no lower than NVIDIA GTX970 or AMD RX Vega M / GH performs the work of the entire pipeline with an output image frequency of 30 fps and a resolution of up to 4K.

Claims

CLAUSES OF THE INVENTION 1. A method for forming a panoramic image, characterized by the fact that: - capturing high-resolution video signal using at least two multifaceted modules, - transmission of the raw video signal to the computing unit, - video signal processing, which uses calculations with a single command stream and multiple data streams; - composition of all video signals into a single video stream in equidistant projection format, obtaining a composite video signal, - each video signal undergoes serial processing with its own unique parameters for each function: correction and compensation of distortions in accordance with the technical features of the optics of the multifaceted module; transformation of coordinate systems in projection with bilinear interpolation; gradient mixing of color channels at the intersections of adjacent angular sections of a spherical surface; image filtering by functions: color palette, brightness and saturation, clarity.

2. The method according to paragraph 1, characterized in that an audio signal is captured and synchronized with a composite video signal.

3. The method according to paragraph 2, characterized in that signal encoding is implemented by transmitting an uncompressed audio and video signal to a selected encoder.

4. The method according to paragraph 3, characterized in that the audio and video signal is transmitted to the Internet.

5. A system for implementing the method according to claim 1, characterized in that it contains a multifaceted unit and a computing unit connected by connectors, the multifaceted unit contains at least two multifaceted modules, each of which contains a lens and a video capture sensor, wherein each multifaceted module is calibrated and has an adjusted lens.

6. The system according to paragraph 5, characterized in that the computing unit contains hardware accelerators.

Images