Virtual reality panoramic dynamic shooting method and system based on multi-lens cooperation

Abstract

The invention relates to the technical field of camera shooting. The virtual reality panoramic dynamic shooting method and system based on multi-lens cooperation are provided, and the method comprises the steps: carrying out the exposure time sequence synchronous control of a multi-lens array, and generating multi-channel image data of synchronous exposure; performing noise feature separation processing on the synchronously exposed multichannel image data to obtain a high-frequency noise component and a brightness component of each lens; constructing a multi-channel noise correlation model, carrying out combined suppression processing on the noise of the overlapped region, and generating a noise-reduced multi-channel image; performing geometric alignment and dynamic splicing processing on the multi-channel image after noise reduction to generate a panoramic dynamic image; and carrying out real-time rendering processing on the panoramic dynamic image, and outputting a low-noise panoramic picture with an adaptive view angle, so as to solve the problems of image dislocation and dynamic object ghosting caused by noise accumulation and time sequence jitter in an overlapping region under multi-lens collaboration.

Description

Technical Field

[0001] The present invention relates to the field of camera technology, and particularly to a virtual reality panoramic dynamic shooting method and system based on multi-lens collaboration. Background Art

[0002] With the rapid development of virtual reality technology, panoramic dynamic shooting systems based on multi-lens collaboration play a key role in fields such as film and television production, virtual tourism, industrial simulation, and metaverse construction. High-quality panoramic dynamic imaging ability is the core foundation for enhancing the user's immersive experience.

[0003] However, in traditional technologies, there are problems such as noise accumulation in the overlapping area under multi-lens collaboration, resulting in a decrease in signal-to-noise ratio, which is particularly serious in low-light scenes. At the same time, there is a problem that software synchronous exposure is difficult to eliminate the timing jitter of multi-lens CMOS sensors, resulting in image time misalignment in dynamic scenes, and aggravated motion blur and stitching artifacts. In addition, there is also a problem that traditional stitching algorithms are only optimized for static scenes, lacking tracking of dynamic objects across lenses, resulting in afterimages or tearing of dynamic objects. Summary of the Invention

[0004] Based on this, it is necessary to provide a virtual reality panoramic dynamic shooting method and system based on multi-lens collaboration for the above technical problems, so as to solve the problems of noise accumulation in the overlapping area under multi-lens collaboration, image misalignment caused by timing jitter, and afterimages of dynamic objects, improve the signal-to-noise ratio in low-light scenes, reduce dynamic stitching delay, and significantly enhance the immersion and interaction fluency of virtual reality scenes.

[0005] In the first aspect, the present application provides a virtual reality panoramic dynamic shooting method based on multi-lens collaboration, including:

[0006] Controlling the exposure timing synchronization of the multi-lens array through a hardware synchronization controller to generate multi-channel image data with synchronous exposure;

[0007] Performing noise feature separation processing on the multi-channel image data with synchronous exposure to obtain the high-frequency noise components and luminance components of each lens;

[0008] Based on the high-frequency noise components and luminance components, constructing a multi-channel noise correlation model to jointly suppress the noise in the overlapping area and generate a multi-channel image after noise reduction;

[0009] Performing geometric alignment and dynamic stitching processing on the multi-channel image after noise reduction to generate a panoramic dynamic image;

[0010] Based on the interaction instructions of the virtual reality device, performing real-time rendering processing on the panoramic dynamic image and outputting a low-noise panoramic picture adapted to the viewing angle.

[0011] Further, perform geometric alignment and dynamic stitching on the denoised multi-channel images to generate a panoramic dynamic image, including:

[0012] Perform projective transformation on the denoised multi-channel images based on the feature point matching algorithm to generate geometrically aligned images;

[0013] Perform fade-in and fade-out fusion on the stitching boundary regions of the geometrically aligned images to obtain a seamlessly stitched intermediate image. The seamlessly stitched intermediate image is the single-frame stitching result of the static scene and retains the temporal dynamic information of the original multi-channel images;

[0014] Based on the seamlessly stitched intermediate image and the temporal frame sequence data in the multi-channel image data, use the optical flow method to predict the motion trajectory of the dynamic object to generate a dynamic compensation result;

[0015] Fuse the dynamic compensation result with the seamlessly stitched intermediate image to generate a panoramic dynamic image with motion compensation.

[0016] Further, based on the seamlessly stitched intermediate image and the temporal frame sequence data in the multi-channel image data, use the optical flow method to predict the motion trajectory of the dynamic object to generate a dynamic compensation result, including:

[0017] Based on the seamlessly stitched intermediate image and the temporal frame sequence data, perform dynamic object motion vector calculation to generate an initial motion vector field;

[0018] Perform trajectory prediction optimization on the initial motion vector field to generate smooth motion trajectory data;

[0019] Perform displacement compensation on the dynamic object according to the smooth motion trajectory data to generate a dynamic compensation result.

[0020] Further, based on the high-frequency noise components and the luminance components, construct a multi-channel noise correlation model to jointly suppress the noise in the overlapping regions to generate denoised multi-channel images, including:

[0021] Calculate based on the high-frequency noise components to obtain a multi-channel covariance matrix, which is used to quantify the correlation intensity between different lens noise channels;

[0022] Allocate noise suppression weights according to the inverse matrix of the multi-channel covariance matrix and jointly attenuate the energy of the high-correlation noise channels to generate denoised multi-channel images.

[0023] Further, based on the interaction instructions of the virtual reality device, perform real-time rendering on the panoramic dynamic image and output a low-noise panoramic view adapted to the viewing angle, including:

[0024] Perform local contrast enhancement on the low-light regions of the panoramic dynamic image to generate a panoramic dynamic image with enhanced details;

[0025] Based on the view switching instruction of the virtual reality device, perform viewport dynamic rendering on the panoramic dynamic image with enhanced details to generate a low-noise panoramic image adapted to the view.

[0026] Furthermore, perform noise feature separation on the multi-channel image data with synchronous exposure to obtain the high-frequency noise components and luminance components of each lens, including:

[0027] Perform luminance channel separation on the multi-channel image data with synchronous exposure to obtain the luminance component;

[0028] Based on the luminance component, perform frequency-domain filtering on the multi-channel image data with synchronous exposure to extract the high-frequency noise component;

[0029] Perform spatio-temporal label annotation on the high-frequency noise component to obtain the high-frequency noise component and the altitude component.

[0030] Furthermore, use a hardware synchronization controller to perform exposure timing synchronization control on the multi-lens array to generate multi-channel image data with synchronous exposure, including:

[0031] Obtain a trigger signal;

[0032] Based on the trigger signal, use the hardware synchronization controller to perform exposure timing synchronization control on the CMOS sensors of the multi-lens sequence, so that the exposure start time and exposure duration of all lenses are consistent, and generate multi-channel image data with synchronous exposure.

[0033] In a second aspect, the present application also provides a virtual reality panoramic dynamic shooting system based on multi-lens collaboration, which includes:

[0034] A multi-lens synchronous exposure control module for performing exposure timing synchronization control on the multi-lens array through a hardware synchronization controller to generate multi-channel image data with synchronous exposure;

[0035] A multi-channel noise feature separation module for performing noise feature separation on the multi-channel image data with synchronous exposure to obtain the high-frequency noise components and luminance components of each lens;

[0036] A multi-channel noise joint suppression module for constructing a multi-channel noise correlation model based on the high-frequency noise components and performing joint suppression on the noise in the overlapping regions to generate a multi-channel image after noise reduction;

[0037] A geometric alignment and dynamic stitching module for performing geometric alignment and dynamic stitching on the multi-channel image after noise reduction to generate a panoramic dynamic image;

[0038] A real - time perspective adaptation rendering module, which is used to perform real - time rendering processing on a panoramic dynamic image based on the interaction instructions of a virtual reality device, and output a low - noise panoramic image adapted to the perspective.

[0039] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the virtual reality panoramic dynamic shooting method based on multi - lens cooperation according to any one of the first aspects of the present application.

[0040] In a fourth aspect, the present application also provides a computer - readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the virtual reality panoramic dynamic shooting method based on multi - lens cooperation according to any one of the first aspects of the present application.

[0041] The technical solution provided by the present application includes the following technical effects: By providing a virtual reality panoramic dynamic shooting method based on multi - lens cooperation, the method includes: performing exposure timing synchronization control on a multi - lens array through a hardware synchronization controller to generate multi - channel image data with synchronous exposure; performing noise feature separation processing on the multi - channel image data with synchronous exposure to obtain the high - frequency noise components and luminance components of each lens; based on the high - frequency noise components and luminance components, constructing a multi - channel noise correlation model to perform joint suppression processing on the noise in the overlapping area to generate a multi - channel image after noise reduction; performing geometric alignment and dynamic stitching processing on the multi - channel image after noise reduction to generate a panoramic dynamic image; performing real - time rendering processing on the panoramic dynamic image based on the interaction instructions of the virtual reality device, and outputting a low - noise panoramic image adapted to the perspective, so as to solve the problems of noise accumulation in the overlapping area under multi - lens cooperation, image misalignment caused by timing jitter, and dynamic object afterimages, improve the signal - to - noise ratio in low - light scenes, reduce the dynamic stitching delay, and significantly enhance the immersion and interaction fluency of the virtual reality scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is a flowchart of the virtual reality panoramic dynamic shooting method based on multi - lens cooperation in an embodiment of the present invention;

[0044] Figure 2 It is a structural diagram of the virtual reality panoramic dynamic shooting system based on multi - lens cooperation in an embodiment of the present invention. Detailed implementation manners

[0045] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The present application provides a virtual reality panoramic dynamic shooting method based on multi-lens collaboration, including:

[0047] S101: Perform exposure timing synchronization control on a multi-lens array through a hardware synchronization controller to generate multi-channel image data with synchronized exposure.

[0048] Specifically, during the shooting process of the multi-lens array, to ensure the precise synchronization of the exposure timings of each lens, a hardware synchronization controller is introduced to uniformly regulate the exposure start time and duration of the multi-lens, avoiding image misalignment and stitching artifacts caused by timing differences. When receiving a shooting trigger signal, the hardware synchronization controller immediately responds and sends a synchronized exposure instruction to each lens in the multi-lens array, enabling all lenses to start exposure at the same moment and end exposure synchronously within a specified time, thereby collecting multi-channel image data with synchronized exposure, laying a foundation for subsequent noise suppression, image stitching, and rendering processing, and effectively improving the quality and consistency of panoramic dynamic images.

[0049] S102: Perform noise feature separation processing on the multi-channel image data with synchronized exposure to obtain the high-frequency noise components and luminance components of each lens.

[0050] Specifically, first, perform luminance channel separation processing on the multi-channel image data with synchronized exposure. By decomposing the image data into different color channels, the luminance-related components, i.e., the luminance components, are extracted. Then, based on the luminance components, perform frequency-domain filtering processing on the multi-channel image data with synchronized exposure. Using frequency-domain analysis methods such as wavelet transform, the image is decomposed into sub-bands of different frequencies, and the high-frequency part is extracted as the high-frequency noise component. Finally, perform spatio-temporal label annotation processing on the high-frequency noise components. According to the spatial distribution and temporal characteristics of the noise in the image, label its position and intensity in different lenses and different frames, thereby obtaining the high-frequency noise components and luminance components of each lens.

[0051] S103: Based on the high-frequency noise components and luminance components, construct a multi-channel noise correlation model to perform joint suppression processing on the noise in the overlapping regions and generate a multi-channel image after noise reduction.

[0052] Specifically, first, perform noise feature separation processing on the multi-channel image data with synchronous exposure to obtain the high-frequency noise components and luminance components of each lens. Then, calculate based on the high-frequency noise components to obtain a multi-channel covariance matrix, which is used to quantify the correlation strength between the noise channels of different lenses. Next, allocate noise suppression weights according to the inverse matrix of the multi-channel covariance matrix, and perform joint attenuation processing on the energy of the high-correlation noise channels, thereby generating a multi-channel image after noise reduction.

[0053] S104: Perform geometric alignment and dynamic stitching processing on the multi-channel image after noise reduction to generate a panoramic dynamic image.

[0054] Specifically, first, perform projective transformation processing on the multi-channel image after noise reduction based on the feature point matching algorithm to generate an image after geometric alignment, aligning the images of different lenses in space, providing a basis for subsequent stitching. Next, perform fade-in and fade-out fusion processing on the stitching boundary region of the geometrically aligned image to obtain a seamlessly stitched intermediate image, which is the single-frame stitching result of the static scene, retains the temporal dynamic information of the original multi-channel image, eliminates stitching traces, and improves the visual consistency of the image. Then, based on the seamlessly stitched intermediate image and the temporal frame sequence data in the multi-channel image data, use the optical flow method to predict the motion trajectory of dynamic objects to generate a dynamic compensation result to compensate for the motion information of dynamic objects that cannot be processed in the static stitching result, realizing accurate tracking and compensation of dynamic objects. Finally, fuse the dynamic compensation result with the seamlessly stitched intermediate image to generate a panoramic dynamic image after motion compensation, enabling the panoramic image to not only be seamlessly stitched in the static scene but also accurately present the motion state of dynamic objects, improving the overall quality and visual effect of the panoramic dynamic image, and enhancing the immersion and interaction fluency of the virtual reality scene.

[0055] S105: Based on the interaction instructions of the virtual reality device, perform real-time rendering processing on the panoramic dynamic image and output a low-noise panoramic view adapted to the viewing angle.

[0056] Specifically, first, the virtual reality device captures the user's interaction instructions, such as head movements, gesture operations, or voice commands, through built-in components such as the attitude detection circuit. The above interaction instructions are received by the processing component and used as the basis for adjusting the rendered image. Then, the rendering engine performs real-time rendering processing on the panoramic dynamic image according to the received interaction instructions. During the rendering process, information such as lighting, materials, and textures is considered to calculate the final image or video. At the same time, in order to ensure the low-noise characteristics of the output image, local contrast enhancement processing is performed on the low-light regions of the panoramic dynamic image to generate a panoramic dynamic image with enhanced details. Finally, according to the viewport switching instruction of the virtual reality device, the panoramic dynamic image with enhanced details is subjected to viewport dynamic rendering processing to generate a low-noise panoramic image adapted to the user's current view, and it is output to the display component of the virtual reality device to provide the user with an immersive visual experience.

[0057] The technical solution provided by this application includes the following technical effects: By providing a virtual reality panoramic dynamic shooting method based on multi-lens cooperation, the method includes: controlling the exposure timing synchronization of the multi-lens array through a hardware synchronization controller to generate multi-channel image data with synchronized exposure; performing noise feature separation processing on the multi-channel image data with synchronized exposure to obtain the high-frequency noise components and luminance components of each lens; based on the high-frequency noise components and luminance components, constructing a multi-channel noise correlation model to jointly suppress the noise in the overlapping region and generate a multi-channel image after noise reduction; performing geometric alignment and dynamic stitching processing on the multi-channel image after noise reduction to generate a panoramic dynamic image; performing real-time rendering processing on the panoramic dynamic image based on the interaction instructions of the virtual reality device and outputting a low-noise panoramic image adapted to the viewport, so as to solve the problems of noise accumulation in the overlapping region under multi-lens cooperation, image misalignment caused by timing jitter, and dynamic object ghosting, improve the signal-to-noise ratio in low-light scenes, reduce the dynamic stitching delay, and significantly enhance the immersion and interaction fluency of the virtual reality scene.

[0058] Further, performing geometric alignment and dynamic stitching processing on the multi-channel image after noise reduction to generate a panoramic dynamic image includes:

[0059] Performing projective transformation processing on the multi-channel image after noise reduction based on the feature point matching algorithm to generate an image after geometric alignment;

[0060] Performing fade-in and fade-out fusion processing on the stitching boundary region of the image after geometric alignment to obtain an intermediate image with seamless stitching. The intermediate image with seamless stitching is the single-frame stitching result of the static scene and retains the timing dynamic information of the original multi-channel image;

[0061] Based on the intermediate image with seamless stitching and the sequential frame sequence data in multi-channel image data, use the optical flow method to predict the motion trajectory of dynamic objects and generate a dynamic compensation result;

[0062] Fuse the dynamic compensation result with the intermediate image with seamless stitching to generate a panoramic dynamic image after motion compensation.

[0063] Specifically, first, use the feature point matching algorithm to perform projective transformation on the denoised multi-channel image to geometrically align the images of different lenses and generate a geometrically aligned image. Then, perform a fade-in and fade-out fusion process on the stitching boundary region of the geometrically aligned image to make the pixel values at the stitching transition smoothly, obtaining an intermediate image with seamless stitching. This intermediate image is the single-frame stitching result of the static scene and retains the temporal dynamic information of the original multi-channel image. Then, based on the intermediate image with seamless stitching and the sequential frame sequence data in the multi-channel image data, use the optical flow method to predict the motion trajectory of dynamic objects and generate a dynamic compensation result. Finally, fuse the dynamic compensation result with the intermediate image with seamless stitching to generate a panoramic dynamic image after motion compensation, enabling the panoramic image to not only be seamlessly stitched in the static scene but also accurately present the motion state of dynamic objects, improving the overall quality and visual effect of the panoramic dynamic image.

[0064] Furthermore, based on the intermediate image with seamless stitching and the sequential frame sequence data in the multi-channel image data, use the optical flow method to predict the motion trajectory of dynamic objects and generate a dynamic compensation result, including:

[0065] Based on the intermediate image with seamless stitching and the sequential frame sequence data, perform dynamic object motion vector calculation processing to generate an initial motion vector field;

[0066] Perform trajectory prediction optimization processing on the initial motion vector field to generate smooth motion trajectory data;

[0067] Perform displacement compensation processing on the dynamic object according to the smooth motion trajectory data to generate a dynamic compensation result.

[0068] Specifically, first, combine the intermediate image with seamless stitching with the sequential frame sequence data in the multi-channel image data, and use the optical flow method to calculate the motion vectors of dynamic objects between consecutive frames to generate an initial motion vector field. Then, perform optimization processing on the initial motion vector field. By analyzing the continuity and consistency of the vector field, remove outliers and noise to generate smooth motion trajectory data. Finally, according to the smoothed motion trajectory data, predict the position of the dynamic object in the current frame and perform displacement compensation processing to ensure the motion coherence and position accuracy of the dynamic object in the panoramic image, thereby generating a dynamic compensation result.

[0069] Furthermore, based on the high-frequency noise components and the luminance components, a multi-channel noise correlation model is constructed to jointly suppress the noise in the overlapping regions, generating a multi-channel image after noise reduction, including:

[0070] Calculations are performed based on the high-frequency noise components to obtain a multi-channel covariance matrix, which is used to quantify the correlation strength between the noise channels of different lenses;

[0071] Noise suppression weights are assigned according to the inverse matrix of the multi-channel covariance matrix, and the energy of the highly correlated noise channels is jointly attenuated to generate a multi-channel image after noise reduction.

[0072] Specifically, first, noise feature separation processing is performed on the multi-channel image data of synchronous exposure to obtain the high-frequency noise components and the luminance components of each lens. This step is the basis of the entire noise reduction process. Only by accurately separating the noise and luminance information can reliable data support be provided for subsequent noise correlation analysis. Then, calculations are performed based on the high-frequency noise components to obtain a multi-channel covariance matrix. This matrix is used to quantify the correlation strength between the noise channels of different lenses. By analyzing the distribution and correlation of the noise among different channels, it is possible to more accurately identify which channels have higher noise correlation, providing a basis for subsequent joint suppression. Then, noise suppression weights are assigned according to the inverse matrix of the multi-channel covariance matrix. For the highly correlated noise channels, larger weights are given for energy attenuation processing, thereby achieving effective joint suppression of the noise in the overlapping regions. This step is the key to the entire noise reduction process. By reasonably allocating weights, the noise can be suppressed targeted while retaining the important information in the image and avoiding the loss of image details caused by excessive noise reduction.

[0073] Finally, after the above processing, a multi-channel image after noise reduction is generated. This image not only reduces the noise interference within each channel but also achieves collaborative suppression of the noise among the channels, making the panoramic dynamic image have a higher signal-to-noise ratio and better visual effects as a whole, laying a good foundation for subsequent processing such as geometric alignment, dynamic stitching, and real-time rendering.

[0074] Furthermore, based on the interaction instructions of the virtual reality device, real-time rendering processing is performed on the panoramic dynamic image to output a low-noise panoramic view adapted to the viewing angle, including:

[0075] Local contrast enhancement processing is performed on the low-light regions of the panoramic dynamic image to generate a panoramic dynamic image with enhanced details;

[0076] Based on the viewing angle switching instructions of the virtual reality device, viewport dynamic rendering processing is performed on the panoramic dynamic image with enhanced details to generate a low-noise panoramic view adapted to the viewing angle.

[0077] Specifically, first, the virtual reality device captures the user's interaction instructions, such as head movements, gesture operations, or voice commands. These instructions are received by the processing element and used as the basis for adjusting the rendered image. Then, local contrast enhancement processing is performed on the low-light regions of the panoramic dynamic image. A mapping function is learned through deep learning technology to map the input low-light image to a high-light space. This mapping function consists of a global curve and a local curve. The global curve is used to control the brightness of the entire image, and the local curve is used to process the details in the image, thereby generating a panoramic dynamic image with enhanced details. Finally, based on the view switching instruction of the virtual reality device, viewport dynamic rendering processing is performed on the panoramic dynamic image with enhanced details. The rendering engine performs real-time rendering on the panoramic dynamic image according to the view switching instruction, considering information such as lighting, materials, and textures, calculates the image adapted to the user's current view, and generates a low-noise panoramic image adapted to the view, which is output to the display component of the virtual reality device to provide the user with an immersive visual experience.

[0078] Furthermore, noise feature separation processing is performed on the multi-channel image data with synchronous exposure to obtain the high-frequency noise components and luminance components of each lens, including:

[0079] Luminance channel separation processing is performed on the multi-channel image data with synchronous exposure to obtain the luminance component;

[0080] Based on the luminance component, frequency-domain filtering processing is performed on the multi-channel image data with synchronous exposure to extract the high-frequency noise components;

[0081] Spatio-temporal label annotation processing is performed on the high-frequency noise components to obtain the high-frequency noise components and height components.

[0082] Specifically, first, luminance channel separation processing is performed on the multi-channel image data with synchronous exposure. By decomposing the image data into different color channels, the luminance-related component, that is, the luminance component, is extracted. Then, based on the luminance component, frequency-domain filtering processing is performed on the multi-channel image data with synchronous exposure. Using frequency-domain analysis methods such as wavelet transform, the image is decomposed into sub-bands of different frequencies, and the high-frequency part is extracted as the high-frequency noise component. Finally, spatio-temporal label annotation processing is performed on the high-frequency noise components. According to the spatial distribution and time characteristics of the noise in the image, the positions and intensities in different lenses and different frames are labeled, thereby obtaining the high-frequency noise components and luminance components of each lens, providing a basis for subsequent noise joint suppression processing.

[0083] Furthermore, the exposure timing synchronization control of the multi-lens array is performed by the hardware synchronization controller to generate multi-channel image data with synchronous exposure, including:

[0084] Obtain the trigger signal;

[0085] Based on the trigger signal, the exposure timing synchronization control process of the CMOS sensors of the multi-lens sequence is carried out through a hardware synchronization controller, so that the exposure start time and exposure duration of all lenses are kept consistent, and multi-channel image data with synchronous exposure is generated.

[0086] Specifically, first, a trigger signal is obtained. This signal can be triggered by a physical button in the device or by a command program in an application. The command can be to capture a video segment or take a photo. Then, based on this trigger signal, the hardware synchronization controller performs the exposure timing synchronization control process on the CMOS sensors of the multi-lens sequence. The hardware synchronization controller ensures that multiple cameras start capturing images at the same time through a synchronization signal generator or a trigger, and at the same time, calibration is performed through timestamp calibration or the Network Time Protocol (NTP) to ensure that the data captured by the cameras has consistent time stamps. During the control process, the MCU (Microcontroller Unit) sends control commands to the internal DSP of the image sensor through the SCCB interface, receives the image data transmitted by the image sensor and controls it to be stored in the corresponding memory, and then sends the image data in the memory to the host computer. The image sensor has two working modes: the master mode and the slave mode. In the slave mode, the MCU sends control commands to the internal control register of the image sensor to generate the slave mode. The image control signals such as reset, snapshot trigger, snapshot start, frame synchronization, horizontal synchronization, and vertical synchronization are generated by the camera control component and sent to the image sensor, and the image sensor controls the acquisition and transmission of images according to the camera control component.

[0087] After the above processing, the exposure start time and exposure duration of all lenses are kept consistent, thereby generating multi-channel image data with synchronous exposure, laying a good foundation for subsequent processing such as noise suppression, image stitching, and real-time rendering, ensuring that the panoramic dynamic image has a higher signal-to-noise ratio and better visual effects as a whole, and enhancing the immersion and interaction fluency of the virtual reality scene.

[0088] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0089] In one embodiment, the present application further provides a virtual reality panoramic dynamic shooting system 200 based on multi-lens collaboration. The system includes:

[0090] A multi-lens synchronous exposure control module 201, which is used to perform exposure timing synchronization control on a multi-lens array through a hardware synchronization controller to generate multi-channel image data with synchronous exposure.

[0091] A multi-channel noise feature separation module 202, which is used to perform noise feature separation processing on the multi-channel image data with synchronous exposure to obtain the high-frequency noise components and luminance components of each lens.

[0092] A multi-channel noise joint suppression module 203, which is used to construct a multi-channel noise correlation model based on the high-frequency noise components and perform joint suppression processing on the noise in the overlapping area to generate a multi-channel image after noise reduction.

[0093] A geometric alignment and dynamic stitching module 204, which is used to perform geometric alignment and dynamic stitching processing on the multi-channel image after noise reduction to generate a panoramic dynamic image.

[0094] A real-time perspective adaptation rendering module 205, which is used to perform real-time rendering processing on the panoramic dynamic image based on the interaction instructions of a virtual reality device and output a low-noise panoramic image adapted to the perspective.

[0095] Specifically, the multi-lens synchronous exposure control module 201 uses the hardware synchronization controller to perform exposure timing synchronization control on the multi-lens array to generate multi-channel image data with synchronous exposure, ensuring that the exposure start time and duration of all lenses are consistent and avoiding image misalignment and stitching artifacts caused by timing differences. The multi-channel noise feature separation module 202 performs noise feature separation processing on the multi-channel image data with synchronous exposure to obtain the high-frequency noise components and luminance components of each lens, providing a basis for subsequent noise suppression processing. The multi-channel noise joint suppression module 203 constructs a multi-channel noise correlation model based on the high-frequency noise components and performs joint suppression processing on the noise in the overlapping area to generate a multi-channel image after noise reduction, effectively reducing noise interference and improving image quality.

[0096] The geometric alignment and dynamic stitching module 204 performs geometric alignment and dynamic stitching on the denoised multi-channel images to generate a panoramic dynamic image. Through the feature point matching algorithm, it conducts projective transformation to geometrically align the images of different lenses, and performs fade-in and fade-out fusion processing on the stitching boundary region to obtain a seamlessly stitched intermediate image. Meanwhile, in combination with the optical flow method, it predicts the motion trajectory of dynamic objects to generate a dynamic compensation result, ensuring the accurate presentation of the panoramic image on static scenes and dynamic objects. The real-time perspective adaptation rendering module 205 performs real-time rendering on the panoramic dynamic image based on the interaction instructions of the virtual reality device, and outputs a low-noise panoramic view adapted to the perspective. By performing local contrast enhancement on the low-light regions of the panoramic dynamic image, it generates a panoramic dynamic image with enhanced details, and conducts viewport dynamic rendering according to the perspective switching instruction to provide users with an immersive visual experience.

[0097] The geometric alignment and dynamic stitching module 204 is also used for:

[0098] Performing projective transformation on the denoised multi-channel images based on the feature point matching algorithm to generate geometrically aligned images;

[0099] Performing fade-in and fade-out fusion processing on the stitching boundary region of the geometrically aligned images to obtain a seamlessly stitched intermediate image. The seamlessly stitched intermediate image is the single-frame stitching result of the static scene and retains the temporal dynamic information of the original multi-channel images;

[0100] Based on the seamlessly stitched intermediate image and the temporal frame sequence data in the multi-channel image data, using the optical flow method to predict the motion trajectory of dynamic objects to generate a dynamic compensation result;

[0101] Fusing the dynamic compensation result with the seamlessly stitched intermediate image to generate a motion-compensated panoramic dynamic image.

[0102] The geometric alignment and dynamic stitching module 204 is also used for:

[0103] Based on the seamlessly stitched intermediate image and the temporal frame sequence data, performing dynamic object motion vector calculation processing to generate an initial motion vector field;

[0104] Performing trajectory prediction optimization processing on the initial motion vector field to generate smooth motion trajectory data;

[0105] Performing displacement compensation on the dynamic objects according to the smooth motion trajectory data to generate a dynamic compensation result.

[0106] The multi-channel noise joint suppression module 203 is also used for:

[0107] Calculations are performed based on high-frequency noise components to obtain a multi-channel covariance matrix, which is used to quantify the correlation strength between different lens noise channels;

[0108] Based on the inverse matrix of the multi-channel covariance matrix, noise suppression weights are assigned to jointly attenuate the energy of highly correlated noise channels to generate a denoised multi-channel image.

[0109] The real-time perspective adaptation rendering module 205 is further configured to:

[0110] Perform local contrast enhancement processing on the low-light regions of the panoramic dynamic image to generate a panoramic dynamic image with enhanced details;

[0111] Based on the perspective switching instruction of the virtual reality device, perform viewport dynamic rendering processing on the panoramic dynamic image with enhanced details to generate a low-noise panoramic view adapted to the perspective.

[0112] The multi-channel noise feature separation module 202 is further configured to:

[0113] Perform luminance channel separation processing on the multi-channel image data with synchronized exposure to obtain a luminance component;

[0114] Based on the luminance component, perform frequency-domain filtering processing on the multi-channel image data with synchronized exposure to extract high-frequency noise components;

[0115] Perform spatio-temporal label annotation processing on the high-frequency noise components to obtain high-frequency noise components and altitude components.

[0116] The multi-lens synchronized exposure control module 201 is further configured to:

[0117] Obtain a trigger signal;

[0118] Based on the trigger signal, perform exposure timing synchronization control processing on the CMOS sensors of the multi-lens sequence through a hardware synchronization controller so that the exposure start time and exposure duration of all lenses are kept consistent to generate multi-channel image data with synchronized exposure.

[0119] In one embodiment, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0120] In one embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0121] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0122] The above-described embodiments merely represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A virtual reality panoramic dynamic shooting method based on multi-lens collaboration, characterized in that: The method comprises: The exposure timing of the multi-lens array is synchronously controlled by a hardware synchronization controller to generate multi-channel image data of synchronous exposure; Performing noise feature separation processing on the synchronously exposed multi-channel image data to obtain a high-frequency noise component and a brightness component of each lens; Based on the high-frequency noise component and the brightness component, a multi-channel noise correlation model is constructed, and the noise in the overlapping area is jointly suppressed to generate a denoised multi-channel image; Performing geometric alignment and dynamic stitching processing on the denoised multi-channel images to generate a panoramic dynamic image; Based on the interactive instructions of the virtual reality device, the panoramic dynamic image is rendered in real time to output a low-noise panoramic picture that adapts to the viewing angle.

2. The virtual reality panoramic dynamic shooting method based on multi-lens collaboration according to claim 1 is characterized in that: The step of geometrically aligning and dynamically stitching the denoised multi-channel images to generate a panoramic dynamic image includes: Performing projection transformation processing on the denoised multi-channel image based on a feature point matching algorithm to generate a geometrically aligned image; Performing a fade-in and fade-out fusion process on the stitching boundary area of ​​the geometrically aligned images to obtain a seamlessly stitched intermediate image, wherein the seamlessly stitched intermediate image is a single-frame stitching result of a static scene and retains the temporal dynamic information of the original multi-channel image; Based on the seamlessly spliced ​​intermediate image and the time-series frame sequence data in the multi-channel image data, the motion trajectory of the dynamic object is predicted and processed using an optical flow method to generate a motion compensation result; The motion compensation result is merged with the seamlessly spliced ​​intermediate image to generate the motion compensated panoramic dynamic image.

3. The virtual reality panoramic dynamic shooting method based on multi-lens collaboration according to claim 2 is characterized in that: The method of predicting the motion trajectory of the dynamic object based on the seamlessly spliced ​​intermediate image and the time-series frame sequence data in the multi-channel image data using an optical flow method to generate a motion compensation result includes: Based on the seamlessly spliced ​​intermediate image and the time-series frame sequence data, a motion vector calculation process of a dynamic object is performed to generate an initial motion vector field; Performing trajectory prediction optimization processing on the initial motion vector field to generate smooth motion trajectory data; Displacement compensation processing is performed on the dynamic object according to the smooth motion trajectory data to generate the dynamic compensation result.

4. The virtual reality panoramic dynamic shooting method based on multi-lens collaboration according to claim 1, characterized in that: The method of constructing a multi-channel noise correlation model based on the high-frequency noise component and the brightness component, performing joint suppression processing on the noise in the overlapping area, and generating a denoised multi-channel image includes: Calculating based on the high-frequency noise component to obtain a multi-channel covariance matrix, wherein the multi-channel covariance matrix is ​​used to quantify the correlation strength between different lens noise channels; Noise suppression weights are allocated according to the inverse matrix of the multi-channel covariance matrix, and energy of high-correlation noise channels is jointly attenuated to generate the denoised multi-channel image.

5. The virtual reality panoramic dynamic shooting method based on multi-lens collaboration according to claim 1, characterized in that: The method of performing real-time rendering processing on the panoramic dynamic image based on the interactive instruction of the virtual reality device and outputting a low-noise panoramic image adapted to the viewing angle includes: Performing local contrast enhancement processing on the low-light area of ​​the panoramic dynamic image to generate a panoramic dynamic image with enhanced details; Based on the perspective switching instruction of the virtual reality device, the detail-enhanced panoramic dynamic image is subjected to viewport dynamic rendering processing to generate the low-noise panoramic picture adapted to the perspective.

6. The virtual reality panoramic dynamic shooting method based on multi-lens collaboration according to claim 1, characterized in that: The performing noise feature separation processing on the synchronously exposed multi-channel image data to obtain the high-frequency noise component and brightness component of each lens includes: Performing brightness channel separation processing on the synchronously exposed multi-channel image data to obtain brightness components; Based on the brightness component, frequency domain filtering is performed on the synchronously exposed multi-channel image data to extract high-frequency noise components; The high-frequency noise component is subjected to spatiotemporal labeling processing to obtain the high-frequency noise component and the height component.

7. The virtual reality panoramic dynamic shooting method based on multi-lens collaboration according to claim 1, characterized in that: The method of performing exposure timing synchronization control on the multi-lens array by using a hardware synchronization controller to generate synchronously exposed multi-channel image data includes: Get the trigger signal; Based on the trigger signal, the exposure timing synchronization control processing of the CMOS sensor of the multi-lens sequence is performed through the hardware synchronization controller, so that the exposure start time and exposure duration of all lenses are consistent, and the synchronously exposed multi-channel image data is generated.

8. A virtual reality panoramic dynamic shooting system based on multi-lens collaboration, characterized in that: The system comprises: A multi-lens synchronous exposure control module is used to perform exposure timing synchronization control on a multi-lens array through a hardware synchronization controller to generate multi-channel image data for synchronous exposure; A multi-channel noise feature separation module is used to perform noise feature separation processing on the synchronously exposed multi-channel image data to obtain a high-frequency noise component and a brightness component of each lens; A multi-channel noise joint suppression module is used to construct a multi-channel noise correlation model based on the high-frequency noise component, perform joint suppression processing on the noise in the overlapping area, and generate a denoised multi-channel image; A geometric alignment and dynamic stitching module, used for performing geometric alignment and dynamic stitching processing on the denoised multi-channel images to generate a panoramic dynamic image; The real-time viewing angle adaptation rendering module is used to perform real-time rendering processing on the panoramic dynamic image based on the interactive instructions of the virtual reality device, and output a low-noise panoramic picture with an adapted viewing angle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the virtual reality panoramic dynamic shooting method based on multi-lens collaboration described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the virtual reality panoramic dynamic shooting method based on multi-lens collaboration described in any one of claims 1 to 7 are implemented.

Images