Panoramic video real-time processing method based on AI

Abstract

The invention relates to the technical field of panoramic video processing, in particular to an AI-based panoramic video real-time processing method. The method comprises the following steps: acquiring video data of multiple cameras, setting geometric correction and color equalization parameters, and preprocessing the video data; performing real-time identification and segmentation on an object in the video data based on an AI model, analyzing dynamic characteristics of the object, and performing compression optimization on the video data to obtain a panoramic video; synchronizing multi-channel data, and outputting the data to terminal equipment in real time; by using consistent geometric correction and color equalization parameters, the multi-channel video data at the same real-time time is processed, so that multidirectional picture hues are kept consistent, adjacent orientation overlapping regions can be accurately identified to facilitate subsequent region fusion, and the watching experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic video processing, and in particular to a panoramic video real-time processing method based on AI. Background Art

[0002] With the rapid development of virtual reality and augmented reality technologies, panoramic videos are being used more and more widely. However, when the traditional panoramic video processing method fuses data from multiple cameras, due to the lighting and camera angle issues in each direction, the subsequent panoramic video will have a color tone inconsistency between the images in one or more directions and the images in other directions. At the same time, the edges of the images in the directions with inconsistent colors overlap with the edges of the images in the adjacent directions. Due to the inconsistent colors, the overlapping area cannot be identified, which affects the user's viewing of the panoramic video.

[0003] Therefore, it is necessary to propose an AI-based real-time panoramic video processing method that can maintain the consistency of multi-directional picture tones and accurately identify the overlapping areas of adjacent directions. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time processing method for panoramic videos based on AI, aiming to solve the problem in the prior art that the color tone of pictures in one or more directions is inconsistent with that of pictures in other directions, and at the same time, the edges of pictures in directions with inconsistent color tone overlap with the edges of pictures in adjacent directions, and the overlapping area cannot be identified due to the inconsistent color tone, which affects the user's viewing of panoramic videos.

[0005] To achieve the above purpose, the present invention adopts an AI-based panoramic video real-time processing method, which includes the following steps:

[0006] Obtain video data from multiple cameras, set geometric correction and color balance parameters, and pre-process the video data;

[0007] Based on the AI ​​model, objects in the video data are identified and segmented in real time, their dynamic features are analyzed, and the video data is compressed and optimized to obtain panoramic video;

[0008] Synchronize multi-channel data and output to terminal devices in real time.

[0009] Among them, in the step of obtaining video data from multiple cameras, setting geometric correction and color balance parameters, and preprocessing the video data:

[0010] Collect video data from multiple cameras, assign real-time time identifiers, and obtain video data from each camera;

[0011] According to the color difference of each channel of video data, geometric correction and color balance parameters are obtained to correct the video data with the same real-time time identifier;

[0012] Concatenate the corrected video data of the same real-time time identifier.

[0013] Among them, in the step of obtaining geometric correction and color balance parameters according to the color difference of each channel of video data, and correcting the video data with the same real-time time identifier:

[0014] Correction methods include image distortion correction, exposure balance, and color adjustment.

[0015] Wherein, in the step of splicing the corrected video data of the same real-time time identifier:

[0016] According to the intersection of the coverage area of ​​each camera, the overlapping area of ​​multiple video data is obtained;

[0017] The overlapping areas are merged to eliminate seams and redundant data and complete the stitching process.

[0018] Among them, in the step of real-time recognition and segmentation of objects in video data based on AI models, analysis of their dynamic characteristics, and compression and optimization of video data to obtain panoramic video:

[0019] Use AI models to analyze the dynamic features of objects in video data, and select and optimize the video frame rate and resolution of video data;

[0020] Obtaining the mapping relationship between low-definition video and high-definition video, generating high-definition video frames from video data, and performing compression optimization;

[0021] The enhanced video frames are reassembled into a complete panoramic video.

[0022] Among them, in the step of using the AI ​​model to analyze the dynamic characteristics of objects in the video data, and selecting and optimizing the video frame rate and resolution of the video data:

[0023] Dynamic features include object movement trajectory and user perspective changes.

[0024] Among them, in the step of synchronizing multi-channel data and outputting it to the terminal device in real time:

[0025] Synchronize multiple channels of data separately, determine the video encoding format based on the transmission restrictions of the terminal equipment, and transmit it.

[0026] After synchronizing multi-channel data and outputting them to the terminal device in real time:

[0027] Dynamically adjust the quality and frame rate of panoramic videos.

[0028] The present invention discloses an AI-based real-time panoramic video processing method, which obtains video data of multiple cameras, sets geometric correction and color balancing parameters, and pre-processes the video data; based on the AI ​​model, objects in the video data are recognized and segmented in real time, their dynamic characteristics are analyzed, and the video data is compressed and optimized to obtain the panoramic video; multi-channel data is synchronized and output to a terminal device in real time; and multiple video data of the same real time are processed by using consistent geometric correction and color balancing parameters to achieve the consistency of the picture tones in multiple directions, accurately identify the overlapping areas of adjacent directions for subsequent area fusion, and improve the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a flowchart of the steps of the AI-based panoramic video real-time processing method of the present invention.

[0031] Figure 2 It is a step flow chart of S100 of the present invention.

[0032] Figure 3 It is a step flow chart of S130 of the present invention.

[0033] Figure 4 It is a step flow chart of S200 of the present invention. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0035] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0037] See also Figure 1 to Figure 4 The present invention provides a panoramic video real-time processing method based on AI, comprising the following steps:

[0038] S100: Acquire video data from multiple cameras, set geometric correction and color balance parameters, and pre-process the video data;

[0039] S200: Based on the AI ​​model, it can identify and segment objects in video data in real time, analyze their dynamic characteristics, and compress and optimize the video data to obtain panoramic video.

[0040] S300: Synchronizes multi-channel data, outputs it to the terminal device in real time, and dynamically adjusts the quality and frame rate of the panoramic video.

[0041] In this embodiment, video data from multiple cameras is first obtained, geometric correction and color balancing parameters are set, and the video data is preprocessed; then, based on the AI ​​model, objects in the video data are recognized and segmented in real time, their dynamic characteristics are analyzed, and the video data is compressed and optimized to obtain a panoramic video; finally, multi-channel data is synchronized and output to the terminal device in real time, and the quality and frame rate of the panoramic video are dynamically adjusted; by using consistent geometric correction and color balancing parameters, multiple video data at the same real time are processed to achieve consistent picture tones in multiple directions, and can accurately identify overlapping areas in adjacent directions for subsequent area fusion, thereby improving the user's viewing experience.

[0042] Furthermore, in the step of obtaining video data from multiple cameras, setting geometric correction and color balance parameters, and preprocessing the video data:

[0043] S110: Collect video data from multiple cameras, assign a real-time time identifier, and obtain video data from each camera;

[0044] S120: According to the color difference of each channel of video data, geometric correction and color balance parameters are obtained, and video data with the same real-time time identifier is corrected; wherein the correction method includes image distortion correction, exposure balance, and color adjustment;

[0045] S130: splicing the corrected video data with the same real-time time identifier.

[0046] In this embodiment, video data from multiple cameras are first collected and assigned real-time time identifiers, and the video data from each camera is obtained respectively; then, according to the color difference of each video data, geometric correction and color balance parameters are obtained, and the video data with the same real-time time identifier is corrected; wherein the correction method includes image distortion correction, exposure balance, and color adjustment; then, the corrected video data with the same real-time time identifier is spliced; using consistent geometric correction and color balance parameters, multiple video data of the same real-time are processed, which can maintain the consistency of picture tones in multiple directions and accurately identify overlapping areas in adjacent directions for subsequent area fusion.

[0047] Further, in the step of splicing the corrected video data of the same real-time identifier:

[0048] S131: Obtain overlapping areas of multiple channels of video data according to the intersection of each camera coverage area;

[0049] S132: The overlapping areas are merged to eliminate the stitching seams and redundant data, and the stitching process is completed.

[0050] In this embodiment, the coverage area of ​​each camera is calculated based on information such as the camera's installation angle, lens focal length, viewing distance, and photosensitive device size. The straight line equation can be used to determine whether there is an intersection between the coverage areas of different cameras, thereby determining the overlapping area. After obtaining the overlapping area of ​​multiple video data, the overlapping area is fused. A weighted smoothing method can be used to weight the grayscale values ​​of the overlapping area and the grayscale values ​​of the corresponding points to eliminate stitching seams and redundant data and complete the stitching process.

[0051] Furthermore, in the step of real-time recognition and segmentation of objects in video data based on the AI ​​model, analysis of dynamic features, and compression and optimization of video data to obtain panoramic video:

[0052] S201: using an AI model to analyze dynamic features of objects in video data, and selecting and optimizing the video frame rate and resolution of the video data; wherein the dynamic features include object movement trajectories and user perspective changes;

[0053] S202: Acquire a mapping relationship between the low-definition video and the high-definition video, generate a high-definition video frame from the video data, and perform compression optimization;

[0054] S203: Reassemble the enhanced video frames into a complete panoramic video.

[0055] In this embodiment, the dynamic features of objects in the video data are first analyzed, and the video frame rate and resolution of the video data are selected and optimized; wherein the dynamic features include the movement trajectory of the object and the change of the user's perspective. Then, the mapping relationship between the low-definition video and the high-definition video is obtained, and the video data is generated into a high-definition video frame, and compression optimization is performed. While ensuring high-quality output, the AI ​​compression algorithm is used to effectively reduce the amount of video data to adapt to the bandwidth requirements under different network environments. Then, the enhanced video frames are recombined into a complete panoramic video, and the dynamic images in the panoramic video are further analyzed and optimized using a deep learning model. The AI ​​model can identify and segment objects in the scene in real time, and analyze dynamic features in the video, such as the trajectory of moving objects, changes in user perspective, etc. By selecting and optimizing the video frames, the balance between the video frame rate and resolution is ensured to reduce delays and improve picture smoothness.

[0056] Furthermore, in the step of synchronizing multi-channel data and outputting it to the terminal device in real time:

[0057] Synchronize multiple channels of data separately, determine the video encoding format based on the transmission restrictions of the terminal equipment, and transmit it.

[0058] In this embodiment, data of multiple channels are synchronized separately, and the video encoding format is determined according to the transmission limitation of the terminal device, and then transmitted; at the same time, the quality and frame rate of the panoramic video are dynamically adjusted according to the bandwidth conditions.

[0059] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0060] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A panoramic video real-time processing method based on AI, characterized in that: The steps include: Obtain video data from multiple cameras, set geometric correction and color balance parameters, and pre-process the video data; Based on the AI ​​model, objects in video data are identified and segmented in real time, and their dynamic characteristics are analyzed. And compress and optimize the video data to obtain panoramic video; Synchronize multi-channel data and output to terminal devices in real time.

2. The AI-based real-time panoramic video processing method according to claim 1, characterized in that: In the steps of obtaining video data from multiple cameras, setting geometric correction and color balance parameters, and preprocessing the video data: Collect video data from multiple cameras, assign real-time time identifiers, and obtain video data from each camera; According to the color difference of each channel of video data, geometric correction and color balance parameters are obtained to correct the video data with the same real-time time identifier; Concatenate the corrected video data of the same real-time time identifier.

3. The AI-based real-time panoramic video processing method according to claim 2, characterized in that: In the step of obtaining geometric correction and color balance parameters according to the color difference of each channel of video data and correcting the video data with the same real-time time identifier: Correction methods include image distortion correction, exposure balance, and color adjustment.

4. The AI-based real-time panoramic video processing method according to claim 2, characterized in that: In the step of splicing the corrected video data of the same real-time identifier: According to the intersection of the coverage area of ​​each camera, the overlapping area of ​​multiple video data is obtained; The overlapping areas are merged to eliminate seams and redundant data and complete the stitching process.

5. The AI-based real-time panoramic video processing method according to claim 1, characterized in that: Based on the AI ​​model, objects in video data are identified and segmented in real time, and their dynamic characteristics are analyzed. And compress and optimize the video data to obtain the panoramic video steps: Use AI models to analyze the dynamic features of objects in video data, and select and optimize the video frame rate and resolution of video data; Obtaining the mapping relationship between low-definition video and high-definition video, generating high-definition video frames from video data, and performing compression optimization; The enhanced video frames are reassembled into a complete panoramic video.

6. The AI-based real-time panoramic video processing method according to claim 5, characterized in that: In the step of using an AI model to analyze the dynamic features of objects in video data and selecting and optimizing the video frame rate and resolution of the video data: Dynamic features include object movement trajectory and user perspective changes.

7. The AI-based real-time panoramic video processing method according to claim 1, characterized in that: In the steps of synchronizing multi-channel data and outputting it to the terminal device in real time: Synchronize multiple channels of data separately, determine the video encoding format based on the transmission restrictions of the terminal equipment, and transmit it.

8. The AI-based real-time panoramic video processing method according to claim 1, characterized in that: After synchronizing multi-channel data and outputting it to the terminal device in real time: Dynamically adjust the quality and frame rate of panoramic videos.

Images