Panoramic video processing system and method based on artificial intelligence

Abstract

The invention aims to provide a panoramic video processing system and method based on artificial intelligence. The method comprises the following steps: extracting a key frame in a panoramic video; redundant information in the key frame is removed; and splicing the key frames from which the redundant information is removed to obtain an optimized panoramic video. The panoramic video is corrected and optimized through artificial intelligence, the defect that the panoramic video cannot meet the requirements of audiences due to insufficient shooting skills of photographers or low equipment hardware configuration is overcome, the complexity of manually processing the panoramic video is avoided, the processing efficiency of the panoramic video is improved, and the user experience is improved. A panoramic video with excellent quality can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a panoramic video processing system and method based on artificial intelligence. Background Art

[0002] Panoramic video is a new technology. It uses 3D cameras to shoot 360-degree videos. It can record everything that happens around a certain range and a certain period of time. The most attractive thing about panoramic video is that its 360-degree images can bring a high sense of immersion to the audience, giving the audience an immersive and realistic experience. When watching panoramic videos, the audience can adjust the video up and down, left and right at will for a full-scale viewing. This technology has actually been around for a long time, but it has only begun to mature in recent years. Panoramic video has four main characteristics: Immersion: Panoramic video provides an immersive experience through 360-degree images, making the audience feel as if they are in the video scene. Free perspective: The audience can freely adjust the perspective and watch the video content from any angle, providing great interactivity and exploration. High information content: Panoramic video provides more information when showing the scene, and through 360-degree space display, the audience can get a richer visual experience. Mature technology: In recent years, with the development of 3D cameras and post-processing technology, the production and viewing experience of panoramic videos has been significantly improved.

[0003] At present, panoramic video shooting has defects due to insufficient shooting skills of photographers or low hardware configuration of equipment, such as: Exposure defects: Panoramic photos may encounter problems in exposure, including large-area overexposure, partial overexposure, partial underexposure, etc. These problems will affect the overall brightness and detail performance of the photo, and thus affect the viewing experience of panoramic videos. Accuracy defects: Insufficient accuracy is another common problem in panoramic video production, which is manifested as blurred images and lack of accuracy. This may be caused by insufficient shooting skills or low hardware configuration of equipment, resulting in the audience being unable to see the details clearly, affecting the viewing experience. Noise defects: Panoramic videos may have large-scale or partial noise, affecting the clarity and viewing quality of the video. Improper handling of sky and ground patching: The sky and ground are not fully patched and obvious traces of patching appear, which not only affects the visual effect of the video, but also may give the audience an unnatural viewing experience. Summary of the invention

[0004] The purpose of the present invention is to provide a panoramic video processing system and method based on artificial intelligence. The method uses artificial intelligence to correct and optimize panoramic videos, making up for the fact that panoramic videos cannot meet the requirements of the audience due to insufficient shooting skills of the photographer or low hardware configuration of the equipment, avoiding the tediousness of manual processing of panoramic videos, improving the efficiency of panoramic video processing, and being able to obtain panoramic videos with excellent quality. The panoramic video processing method based on artificial intelligence includes: Extract key frames from panoramic videos; removing redundant information in the key frame; The key frames after removing redundant information are spliced ​​to obtain an optimized panoramic video.

[0005] Preferably, extracting key frames from the panoramic video includes: Preprocess the panoramic video and filter the candidate videos, which can be expressed as: Where n is the number of clips in the panoramic video; Combine all the images from the first frame to the last frame in I into two adjacent frames, divide them into a×a blocks, and calculate the mutual information entropy between the sub-image blocks; The mutual information entropy between each frame of the image is inversely calculated to obtain the entropy value ; The entropy values ​​in each frame are added together to calculate the difference between two adjacent frames in the panoramic video, which is expressed as: if If it exceeds the preset value, it means that the similarity between two adjacent frames in the panoramic video is high and is marked as a mutation frame.

[0006] Preferably, extracting key frames from the panoramic video includes: Set the judgment function to determine whether it is a key frame, expressed as: in, is the time domain feature difference curve of the key image frame in the panoramic video, M represents the number of qualified image frames contained in the panoramic video, and K represents the pixel difference of the key image frame.

[0007] Preferably, the removing redundant information in the key frame comprises: Performing feature extraction on the key frame; Cluster and label the extracted features; A neural network model is used to remove redundant information in key frames according to the categories of key frames.

[0008] Preferably, said extracting features from said key frame comprises: Extract texture features from keyframes: in, is the gradient amplitude of the pixel at (x, y), is the gradient direction of the pixel at (x, y), Represents the horizontal gradient value at the pixel point (x, y). Represents the vertical gradient value at the pixel point (x, y); Extract color features from keyframes: in, represents the i-th color component of the j-th pixel in the panoramic video, and N represents the number of pixels in a frame of video image.

[0009] Preferably, clustering and labeling the extracted features includes: Use the roulette wheel method to calculate cluster centers: Among them, P is the probability that each sample i is selected as the next cluster center, and D(i) is the distance between the current sample and the nearest cluster center; The roulette wheel method is used to select K cluster centers; Calculate the distance between each sample and the K cluster centers and assign it to the class corresponding to the cluster center with the smallest distance; For each category , recalculate the cluster center of each category: Repeat the calculation until the position of the cluster center no longer changes, and obtain a total of K cluster centers and the feature vector categories corresponding to the image; The K categories are marked and the processed panoramic video frame is represented as I u .

[0010] Preferably, the panoramic video obtained by splicing key frames after removing redundant information to obtain an optimized panoramic video includes: The image fusion algorithm is used to merge and splice the overlapping areas of adjacent frames line by line. The fused image is expressed as: in, is the weight gradient factor, is the processed panoramic video frame, is the pixel point in the processed panoramic video frame, is the pixel point in the stitched panoramic video frame; in, is the number of overlapping pixels in the yth row, The coordinates of the boundary of the overlapping area in row y; Optimized panoramic video is obtained through frame-by-frame fusion and stitching.

[0011] The AI-based panoramic video processing system includes: A key frame extraction module, used to extract key frames in panoramic videos; A key frame processing module, used to remove redundant information in the key frame; The stitching module is used to stitch the key frames after removing redundant information to obtain an optimized panoramic video.

[0012] An electronic device includes: a processor and a memory, wherein the memory is used to store computer program codes, and the computer program codes include computer instructions. When the processor executes the computer instructions, the electronic device executes a panoramic video processing method based on artificial intelligence.

[0013] A computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes a panoramic video processing method based on artificial intelligence.

[0014] The beneficial effects of the present invention are: 1. In order to improve the processing capabilities of panoramic video lens edge detection and key frame extraction, the present invention proposes a panoramic video key frame data real-time extraction system design based on mutual information entropy. The present invention can accurately extract the key frame data of the panoramic monitoring video, has better panoramic video processing capabilities, and the use of key frame data extraction can reduce the amount of data generated in panoramic video retrieval; 2. In view of the problem of a large amount of information redundancy in panoramic videos, the present invention then uses the HOG+HSV color space first-order moment feature to extract the key frame feature vector, and then uses the K-means++ clustering algorithm to cluster the feature vectors and realize the marking of the key frames, thereby removing redundant information in the key frames; 3. The present invention realizes seamless splicing of panoramic videos through a row-by-row weighted fusion algorithm and image straightening. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor. Figure 1 This is a flow chart of a panoramic video processing method based on artificial intelligence of the present invention; Figure 2 It is a schematic diagram of the process of removing redundant information in key frames according to the present invention; Figure 3 It is a schematic diagram of the neural network structure of the present invention; Figure 4 The figure is a schematic diagram of the hardware structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0018] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0019] At present, panoramic video shooting has defects due to insufficient shooting skills of photographers or low hardware configuration of equipment, such as: Exposure defects: Panoramic photos may encounter problems in exposure, including large-area overexposure, partial overexposure, partial underexposure, etc. These problems will affect the overall brightness and detail performance of the photo, and thus affect the viewing experience of panoramic videos. Accuracy defects: Insufficient accuracy is another common problem in panoramic video production, which is manifested as blurred images and lack of accuracy. This may be caused by insufficient shooting skills or low hardware configuration of equipment, resulting in the audience being unable to see the details clearly, affecting the viewing experience. Noise defects: Panoramic videos may have large-scale or partial noise, affecting the clarity and viewing quality of the video. Improper handling of sky and ground patching: The sky and ground are not fully patched and obvious traces of patching appear, which not only affects the visual effect of the video, but also may give the audience an unnatural viewing experience.

[0020] In order to improve the processing capabilities of panoramic video lens edge detection and key frame extraction, the present invention proposes a panoramic video key frame data real-time extraction system design based on mutual information entropy. The present invention can accurately extract the key frame data of the panoramic monitoring video, has better panoramic video processing capabilities, and the use of key frame data extraction can reduce the amount of data generated in panoramic video retrieval; the present invention aims at the problem of a large amount of information redundancy in the panoramic video, and then uses the HOG+HSV color space first-order moment feature to extract the key frame feature vector, and then uses the K-means++ clustering algorithm to cluster the feature vector and realize the marking of the key frame, so as to remove the redundant information in the key frame; the present invention realizes the seamless splicing of the panoramic video through a row-by-row weighted fusion algorithm and an image straightening method.

[0021] Example 1 Panoramic video processing method based on artificial intelligence, reference Figure 1 , including the steps of: Step S100, extracting key frames in the panoramic video; ‌‌Keyframes of panoramic videos‌ are an important concept in video production. They refer to frames at a specific time point during the video editing process. This frame records all the information of the video at that time point, including picture content, special effects, audio, etc. The role of keyframes is that they fix all the operation information at a specific time point. If you need to adjust the picture or other parameters at this time point, you can do so by updating the position of the keyframe. In the editing of panoramic videos, keyframes are widely used, including but not limited to controlling the volume, achieving lens push-pull effects, and creating color change effects. Control volume‌: By adding keyframes to the audio, you can adjust the volume at different time points to achieve volume changes. This is especially important in videos with double-layer audio tracks, which can prevent the background music from being too loud and covering the original sound, thereby affecting the video effect. Lens push-pull effect‌: By setting keyframes at different positions in the video, you can control the zooming in and out of the picture, simulating the visual effect of the camera pushing in or out, and enhancing the dynamic and immersive feeling of the video. Color change effect: By setting keyframes at different time points in the video, you can control the color change of the picture, achieve the effect of gradual change from one color to another, and increase the visual appeal of the video. The use of keyframes is not limited to panoramic videos. It is an important tool in all video editing. By carefully setting keyframes, the expressiveness and viewing experience of the video can be greatly enhanced. For example, in video editing, keyframes can be used to achieve various dynamic effects, such as movement, scaling, and rotation of objects. In video post-production, keyframes are also used to achieve complex camera effects, video transition effects, etc. By setting keyframes, you can flexibly control the dynamic effects of the video to make the video more vivid and attractive.

[0022] Step S200, removing redundant information in the key frame; The redundant information in key frames mainly includes temporal redundancy, spatial redundancy, coding redundancy and visual redundancy. Temporal redundancy: refers to the similarity of content between two adjacent frames in a video, and the existence of a motion relationship. This redundancy allows the video coding standard algorithm to deal with it through special technologies and improve the compression ratio from different angles. Spatial redundancy: occurs between adjacent pixels within a frame in a video, and the redundancy is caused by the similarity between them. This is also a type of redundancy that is considered and attempted to be reduced in video coding standards. Coding redundancy: involves the different probabilities of different data appearing in the video. The coding standard uses different coding technologies to handle this redundancy to improve compression efficiency. Visual redundancy: is related to the different sensitivities of the audience's visual system to different parts of the video. The coding standard uses different technologies to deal with this visual redundancy to optimize the compression ratio of the video.

[0023] Step S300, stitching the key frames after removing redundant information to obtain an optimized panoramic video.

[0024] ‌Keyframe splicing‌ refers to the splicing and transition of video effects through keyframes in video editing. Keyframe is an important concept in video production. It refers to the key picture at a certain time point in the video production process. The changes in this picture will be recorded and used as a reference point in the subsequent video production process. By setting keyframes at different time points, the smooth transition of effects such as object movement and picture changes in the video can be achieved. The application of keyframe splicing includes but is not limited to controlling the volume, achieving lens push-pull effects, and making electronic photo albums. For example, in the scene of controlling the volume, by setting keyframes on the audio, the volume change can be finely controlled, so as to achieve the gradual change effect of audio in the video. In the lens push-pull effect, by setting keyframes at different time points in the video, the lens can be smoothly advanced or pulled away, enhancing the visual effect of the video. The addition and operation of keyframes are usually carried out in professional video editing software. These software provide intuitive interfaces and tools, allowing users to easily add, delete and adjust keyframes to achieve various complex video effects. By splicing key frames, video makers can create dynamic visual effects and enhance the viewing and artistic quality of the video.

[0025] Preferably, step S100, extracting key frames from the panoramic video includes: Preprocess the panoramic video and filter the candidate videos, which can be expressed as: Where n is the number of clips in the panoramic video; Combine all the images from the first frame to the last frame in I into two adjacent frames, divide them into a×a blocks, and calculate the mutual information entropy between the sub-image blocks; Mutual information entropy is a measure of the correlation between two random variables. It indicates how much the uncertainty of one random variable is reduced when the other random variable is known.

[0026] The mutual information entropy between each frame of the image is inversely calculated to obtain the entropy value ; The entropy values ​​in each frame are added together to calculate the difference between two adjacent frames in the panoramic video, which is expressed as: if If it exceeds the preset value, it means that the similarity between two adjacent frames in the panoramic video is high and is marked as a mutation frame.

[0027] Mutual information entropy is an important indicator to measure the amount of information that a random variable tells about another random variable. It reflects the degree to which the uncertainty of another random variable is reduced when one random variable is known. In information theory, the calculation of mutual information entropy requires the discretization of variables X and Y, which involves how to efficiently bin the data to calculate the correlation between the two variables. By calculating mutual information entropy, the efficiency of information transmission and conversion in a system or process can be evaluated.

[0028] In an embodiment of the present invention, two frames of images to be detected are randomly divided into sub-image blocks, and the mutation frame of the panoramic video image is obtained by calculating the mutual information entropy between the sub-image blocks. The characteristic difference curve of the key image frame is used as the timing feature of the panoramic video key frame to complete the feature matching of the key frame data in the panoramic video. A feature that can reflect the attributes of the panoramic video image is selected to describe the main information in the video. The features of the panoramic video frame are converted into feature vectors in the form of an array by using a numerical analysis method. The number of key frames of the panoramic video is extracted by calculating the mutual information entropy value; thereby, the key frame data of the panoramic video can be accurately extracted, and the panoramic video processing capability is better.

[0029] Preferably, step S100, extracting key frames from the panoramic video includes: Set the judgment function to determine whether it is a key frame, expressed as: in, is the time domain feature difference curve of the key image frame in the panoramic video, M represents the number of qualified image frames contained in the panoramic video, and K represents the pixel difference of the key image frame.

[0030] If the pixel difference of the key image frame is within the tolerance range of the judgment function and is smaller than the threshold set by the user, it means that the features of the key frame data in the panoramic video are successfully matched and marked as a key frame.

[0031] Preferably, reference Figure 2 and Figure 3 , step S200, removing redundant information in the key frame includes: Step S210, extracting features from key frames; ‌Keyframe feature extraction‌ refers to extracting the most important and representative image frames from a video that represent a shot. These frames contain the main information of the shot and can accurately describe the changes in the video content. The extraction of key frames is of great significance for video retrieval, analysis and understanding, because it can significantly reduce the time and resources required for video processing, while improving the efficiency and accuracy of video processing. The extraction of key frames is based on the hierarchical structure of the video, that is, the logical units of scenes, shots and frames. Each video consists of a series of related shots, and each shot consists of a number of continuous frames. Keyframe extraction technology identifies those frames that best represent the content of the shot by analyzing the changes and correlations between video frames. These key frames can not only reduce the redundancy of video data, but also reduce the complexity of data processing and analysis while maintaining the integrity of the video content.

[0032] Step S220, clustering and labeling the extracted features; ‌Feature clustering‌ is a data analysis and machine learning technique that achieves clustering by calculating distances or correlations based on the feature vector matrix of an object. This method focuses on dividing a feature set into multiple clusters based on the correlations between features and between features and feature clusters. Feature clustering does not directly process raw data, but rather processes the features of the data, which can be numerical or categorical, and together constitute feature vectors. By calculating the distance or correlation between these feature vectors, similar features can be clustered together to form different clusters or categories. This technique is very useful for data preprocessing and feature selection, helping us discover patterns and structures in data, simplify and compress data, and can even be used in fields such as prediction and recommendation systems and natural language processing. The process of feature clustering can involve a variety of algorithms, including but not limited to hierarchical clustering, K-means clustering, DBSCAN, etc. These algorithms divide the data set into different clusters or categories based on different metrics (such as Euclidean distance, Mahalanobis distance, etc.), so that the samples within the cluster are as similar as possible, while the samples between different clusters are as different as possible. This clustering method does not require the specific category of the data to be known in advance, so it belongs to a type of unsupervised learning.

[0033] Step S230: Using a neural network model to remove redundant information in the key frames according to the categories of the key frames.

[0034] ‌Neural network removes redundant information in key frames by reducing the temporal redundancy of the video. ‌This method focuses the calculation on some key frames of the video, thereby effectively reducing the computational overhead and improving the efficiency of video recognition. Specifically, by focusing on the key areas in the video frame, that is, the key areas in the key frames, the spatial redundancy is reduced. This method is effectively complementary to the reduction of temporal redundancy, and can simultaneously model spatiotemporal redundancy, thereby focusing on the key areas in the key frames and improving the efficiency and accuracy of video recognition‌.

[0035] Preferably, step S210, extracting features from key frames includes: Extract texture features from keyframes: in, is the gradient amplitude of the pixel at (x, y), is the gradient direction of the pixel at (x, y), Represents the horizontal gradient value at the pixel point (x, y). Represents the vertical gradient value at the pixel point (x, y); The texture characteristics of panoramic videos are mainly reflected in their ability to provide a full range of viewing angles, allowing viewers to watch dynamic videos at any angle within the 360-degree range of the shooting angle. The texture characteristics of panoramic videos include not only visual continuity and consistency, but also the comprehensive expression of multimedia elements such as depth of field, dynamic images, and sound, thereby achieving the effects of sound and picture alignment and synchronization, and enhancing the expressiveness and appeal of the content.

[0036] In the process of panoramic video processing and encoding, by analyzing the texture characteristics of image blocks, the number of candidate modes for intra-frame coding can be reduced, thereby reducing redundant calculations in mode selection and improving coding efficiency.

[0037] Extract color features from keyframes: in, represents the i-th color component of the j-th pixel in the panoramic video, and N represents the number of pixels in a frame of video image.

[0038] The color characteristics of panoramic videos mainly include hue, brightness and saturation. Hue: Hue is the characteristic that distinguishes colors from each other. It is determined by the color spectrum composition of the light source and the feeling of each wavelength emitted by the surface of the object on the human eye. It can distinguish red, yellow, green, blue, purple and other characteristics. In panoramic videos, the adjustment of hue is crucial to creating a specific atmosphere and emotion. Brightness: Brightness is also called brightness. It is a characteristic value that indicates the change in the degree of light and dark on the surface of an object. When the brightness is high, the color gives people a clean and clear feeling; when the brightness is low, the color is more chaotic. In the color adjustment process of panoramic videos, controlling the brightness of the color is of great significance for highlighting the subject and enhancing the visual effect. Saturation: Saturation, also known as chroma, is a characteristic value that indicates the color intensity of the surface of an object, which makes the color bright and dark. If the saturation is too small, the color is dim and the picture lacks color impact; if the saturation is too large, although it can make the color more eye-catching, it will also cause obvious noise in the dark color. In the production of panoramic videos, appropriate saturation adjustment can ensure the vividness of the color and the visual effect of the picture.

[0039] Preferably, step S220, clustering and labeling the extracted features includes: Use the roulette wheel method to calculate cluster centers: Among them, P is the probability that each sample i is selected as the next cluster center, and D(i) is the distance between the current sample and the nearest cluster center; The roulette wheel method is used to select K cluster centers; Calculate the distance between each sample and the K cluster centers and assign it to the class corresponding to the cluster center with the smallest distance; For each category , recalculate the cluster center of each category: Repeat the calculation until the position of the cluster center no longer changes, and obtain a total of K cluster centers and the feature vector categories corresponding to the image; The K categories are marked and the processed panoramic video frame is represented as I u .

[0040] The advantages of the K-means clustering algorithm include simplicity and efficiency, high efficiency and scalability for large data sets, good tolerance to noisy data, clear spatial division, and applicability to large data sets. ‌Simple and efficient‌: The idea of ​​the K-means algorithm is simple and clear, easy to understand and implement. Its calculation process is relatively direct, so it has high execution efficiency‌. High efficiency and scalability for large data sets‌: The K-means algorithm maintains scalability and high efficiency when processing large data sets, and is suitable for clustering tasks of large data sets‌14. Good tolerance to noisy data‌: The K-means algorithm has a certain tolerance for noisy data and can adapt to data with different noise levels by adjusting the number of clusters‌2. Clear spatial division‌: The K-means algorithm can divide the data set into k clear regions by calculating the cluster centers. The data points within each region are highly similar, while the data points between different regions are significantly different. This clear division is helpful for subsequent data analysis and processing‌. Applicable to large data sets‌: Since the calculation process of the K-means algorithm is relatively simple, it is suitable for processing large data sets. Through reasonable optimization and parallel processing, the K-means algorithm can complete the clustering task of a large amount of data in a shorter time, and is very suitable for processing panoramic video feature classification with a large amount of data.

[0041] Preferably, step S300, stitching the key frames after removing redundant information to obtain an optimized panoramic video includes: The image fusion algorithm is used to merge and splice the overlapping areas of adjacent frames line by line. The fused image is expressed as: in, is the weight gradient factor, is the processed panoramic video frame, is the pixel point in the processed panoramic video frame, is the pixel point in the stitched panoramic video frame; in, is the number of overlapping pixels in the yth row, The coordinates of the boundary of the overlapping area in row y; Optimized panoramic video is obtained through frame-by-frame fusion and stitching.

[0042] Panoramic video stitching is a technology that seamlessly stitches together video clips shot by multiple cameras to form a 360-degree panoramic video. The implementation process of panoramic video stitching includes several key steps: Camera calibration: Since the positions and orientations of different cameras may be different, they need to be calibrated to determine the relative positions and orientations between the cameras. Image matching: The captured images are registered using the parameters obtained through camera calibration. This step usually uses feature point extraction and matching methods to find corresponding points between adjacent frames. Image fusion: After obtaining the corresponding points of adjacent frames, they need to be fused to generate a panoramic image or panoramic video. This step usually uses projection mapping or stereo mapping methods to stitch the images of adjacent frames together. Video compression: Since the amount of data in panoramic videos is large, they need to be compressed.

[0043] ‌Panoramic video frame-by-frame fusion is achieved through frame-by-frame stitching to obtain a 360-degree panoramic video or even a 360-degree × 180-degree spherical panoramic video. This technology involves frame-by-frame processing and stitching of video images captured by multiple cameras to achieve a panoramic effect. Specifically, panoramic video stitching technology includes multiple steps: Video acquisition: Use multiple cameras to capture videos of the same scene from different angles. Frame-by-frame stitching: Process the captured video frame by frame, and use algorithms to accurately align and fuse video frames from different perspectives. Generate panoramic video: Through the above steps, a panoramic video covering a 360-degree perspective is finally generated.

[0044] Example 2 The AI-based panoramic video processing system includes: A key frame extraction module, used to extract key frames in panoramic videos; A key frame processing module is used to remove redundant information in key frames; The stitching module is used to stitch the key frames after removing redundant information to obtain an optimized panoramic video.

[0045] Example 3 An electronic device includes: a processor and a memory, the memory is used to store computer program codes, the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes a panoramic video processing method based on artificial intelligence.

[0046] refer to Figure 4, the electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines, or buses, etc., which are not limited in the embodiments of the present invention. It should be understood that in various embodiments of the present invention, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, through various interfaces, transmission lines, buses, etc.

[0047] The processor 21 may be one or more graphics processing units (GPUs). When the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Optionally, the processor 21 may be a processor group consisting of multiple GPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present invention.

[0048] The memory 22 can be used to store computer program instructions and various computer program codes including program codes for executing the scheme of the present invention. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0049] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The output device 24 and the input device 23 may be independent devices or an integrated device.

[0050] Example 4 A computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes a panoramic video processing method based on artificial intelligence.

[0051] In order to improve the processing capabilities of panoramic video lens edge detection and key frame extraction, the present invention proposes a panoramic video key frame data real-time extraction system design based on mutual information entropy. The present invention can accurately extract the key frame data of the panoramic monitoring video, has better panoramic video processing capabilities, and the use of key frame data extraction can reduce the amount of data generated in panoramic video retrieval; the present invention aims at the problem of a large amount of information redundancy in the panoramic video, and then uses the HOG+HSV color space first-order moment feature to extract the key frame feature vector, and then uses the K-means++ clustering algorithm to cluster the feature vector and realize the marking of the key frame, so as to remove the redundant information in the key frame; the present invention realizes the seamless splicing of the panoramic video through a row-by-row weighted fusion algorithm and an image straightening method.

[0052] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A panoramic video processing method based on artificial intelligence, characterized in that: include: Extract key frames from panoramic videos; removing redundant information in the key frame; The key frames after removing redundant information are spliced ​​to obtain an optimized panoramic video.

2. The panoramic video processing method based on artificial intelligence according to claim 1, characterized in that: The extracting of key frames in the panoramic video comprises: Preprocess the panoramic video and filter the candidate videos, which can be expressed as: ， Where n is the number of clips in the panoramic video; Combine all the images from the first frame to the last frame in I into two adjacent frames, divide them into a×a blocks, and calculate the mutual information entropy between the sub-image blocks; The mutual information entropy between each frame of the image is inversely calculated to obtain the entropy value ; The entropy values ​​in each frame are added together to calculate the difference between two adjacent frames in the panoramic video, which is expressed as: ， if If it exceeds the preset value, it means that the similarity between two adjacent frames in the panoramic video is high and is marked as a mutation frame.

3. The panoramic video processing method based on artificial intelligence according to claim 2, characterized in that: The extracting of key frames in the panoramic video comprises: Set the judgment function to determine whether it is a key frame, expressed as: ， in, is the time domain feature difference curve of the key image frame in the panoramic video, M represents the number of qualified image frames contained in the panoramic video, and K represents the pixel difference of the key image frame.

4. The panoramic video processing method based on artificial intelligence according to claim 1, characterized in that: The removing of redundant information in the key frame comprises: Performing feature extraction on the key frame; Cluster and label the extracted features; A neural network model is used to remove redundant information in key frames according to the categories of key frames.

5. The method for processing panoramic video based on artificial intelligence according to claim 4, characterized in that: The feature extraction of the key frame comprises: Extract texture features from keyframes: ， in, is the gradient amplitude of the pixel at (x, y), is the gradient direction of the pixel at (x, y), Represents the horizontal gradient value at the pixel point (x, y). Represents the vertical gradient value at the pixel point (x, y); Extract color features from keyframes: ， in, represents the i-th color component of the j-th pixel in the panoramic video, and N represents the number of pixels in a frame of video image.

6. The method for processing panoramic video based on artificial intelligence according to claim 4, characterized in that: The clustering and labeling of the extracted features includes: Use the roulette wheel method to calculate cluster centers: ， Among them, P is the probability that each sample i is selected as the next cluster center, and D(i) is the distance between the current sample and the nearest cluster center; The roulette wheel method is used to select K cluster centers; Calculate the distance between each sample and the K cluster centers and assign it to the class corresponding to the cluster center with the smallest distance; For each category , recalculate the cluster center of each category: ， Repeat the calculation until the position of the cluster center no longer changes, and obtain a total of K cluster centers and the feature vector categories corresponding to the image; The K categories are marked and the processed panoramic video frame is represented as I u .

7. The method for processing panoramic video based on artificial intelligence according to claim 1, characterized in that: The panoramic video obtained by splicing key frames after removing redundant information to obtain an optimized panoramic video includes: The image fusion algorithm is used to merge and splice the overlapping areas of adjacent frames line by line. The fused image is expressed as: ， in, is the weight gradient factor, is the processed panoramic video frame, is the pixel point in the processed panoramic video frame, is the pixel point in the stitched panoramic video frame; ， in, is the number of overlapping pixels in the yth row, The coordinates of the boundary of the overlapping area in row y; Optimized panoramic video is obtained through frame-by-frame fusion and stitching.

8. The panoramic video processing system based on artificial intelligence is characterized by: include: A key frame extraction module, used to extract key frames in panoramic videos; A key frame processing module, used to remove redundant information in the key frame; The stitching module is used to stitch the key frames after removing redundant information to obtain an optimized panoramic video.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the panoramic video processing method based on artificial intelligence as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes the artificial intelligence-based panoramic video processing method described in any one of claims 1 to 7.

Images