Panoramic video navigation method driven by subjective preference of user

Abstract

The invention discloses a panoramic video navigation method driven by subjective preference of a user, which comprises the following steps: firstly, preprocessing an input panoramic video, extracting high-level semantic information by using a large language model and generating text description, then extracting keywords from the generated text, generating a console after the user selects a plurality of keywords, and carrying out navigation on the console; the preferences are described by adjusting a console. The method comprises the following steps: (1) firstly, a system performs semantic analysis on each panoramic environment based on a lens unit, identifies interested contents, generates guidance according to subjective preferences of a user and movie style specifications, and definitely outputs a semantic focus and a movement type of a lens; and 2) according to the generated guidance, the system uses the control parameters of the lens level to create video candidates for each lens so as to follow the view projection of the shooting rule. And 3) finally, the system aggregates the normal view angle lenses of the projection according to local and global constraints, and combines external knowledge learned from demonstration videos and professional shooting rules.

Description

Technical Field

[0001] The present invention belongs to the technical field of panoramic video navigation methods, and in particular relates to a panoramic video navigation method driven by user subjective preferences. Background Art

[0002] In recent years, with the rapid development of virtual reality (VR) technology and related equipment, panoramic videos have become increasingly popular due to their unique ability to record all directions simultaneously. Today, there are three common ways to watch such videos. The first and most common way is to wear a head-mounted display (HMD). This method can provide viewers with an immersive experience, however, it also requires viewers to constantly rotate their heads and bodies to view different perspectives, which increases the burden on viewers to a certain extent, especially when watching for a long time, which may cause fatigue. The second method is to directly watch the entire spherical video, such as using equirectangular projection (ERP) to unfold the panoramic video. This method allows viewers to see the complete content in each frame. However, since the projection of ERP from the equator to the poles will gradually become severely distorted, the viewing experience becomes less than ideal, especially in the polar regions, where the picture may appear abnormally distorted. The third method is to project a certain area in the ERP as a distortion-free narrow field of view (NFOV). Many video players and websites, such as YouTube and Facebook, use this method to play panoramic videos, allowing viewers to easily watch them using 2D devices such as mobile phones and computers. This method allows viewers to freely change the viewing direction by sliding a mouse or finger, but this manual selection of viewing angles increases the viewer's operational burden to a certain extent.

[0003] With the rapid development of artificial intelligence and deep learning technologies, automated panoramic video navigation methods based on computer vision and machine learning have gradually attracted attention. These methods train models to learn to extract features from video frames and automatically select the best viewport for each video frame, thus providing new possibilities for burden-free viewing of panoramic videos and making the viewing experience closer to that of ordinary 2D videos.

[0004] However, existing panoramic video navigation methods still have some limitations. First, most panoramic video navigation models based on saliency detection usually only focus on the most salient area in the current video frame, lacking a deep understanding of the overall content of the video. This method often generates navigation trajectories that are only local optimal solutions and fail to fully reflect the core content of the entire video. Second, the navigation results generated by existing methods are relatively simple and lack diversity.

[0005] Finally, and most importantly, even if the navigation result predicted by the model does reflect the main content of the entire video, it may not be what all viewers expect to see. Therefore, the most important goal of panoramic video navigation is to make the generated navigation result meet the personalized needs of viewers as much as possible. This requires paying more attention to the interests and preferences of the audience during model design and training, and providing a more intelligent and humanized viewing experience. Summary of the invention

[0006] The present invention aims to solve the technical problems existing in the background technology and to provide a panoramic video navigation method driven by user subjective preferences. The method of the present invention provides a new panoramic video navigation method by combining the personal preferences pre-entered by the user and integrating global and local features. Compared with the existing methods, the method of the present invention not only takes into account the high-level information of the overall content of the video, but also can meet the user's viewing intention as much as possible, thereby improving the accuracy and humanization of the navigation results. In addition, the performance and accuracy of the model can be evaluated and optimized through an independent validation set or test set to improve the robustness and reliability of the model.

[0007] In order to solve the technical problem, the technical solution of the present invention is:

[0008] A panoramic video navigation method driven by user subjective preference, comprising: inputting an ERP color video frame in RGB format into a trained panoramic video navigation model driven by user subjective preference, and obtaining an optimal viewport position corresponding to each frame by the panoramic video navigation model driven by user subjective preference, wherein the construction of the panoramic video navigation model driven by user subjective preference includes:

[0009] S1: Obtain multiple panoramic videos in ERP format and perform preprocessing to obtain training sets and test sets;

[0010] S2: Use the preset video language model to perform preliminary semantic analysis on the images in the training set, extract high-level semantic information of the entire video, and generate text descriptions based on the extracted semantic information;

[0011] S3: Use the fine-tuned language model to extract key semantic information from the text description of the video as the keywords to be selected later;

[0012] S4: Build a graphical user interface that allows users to upload local videos and display the extracted keywords. Users select multiple keywords and set weights, which are then fed into the deep neural network model for processing. It also provides playback control functions, including adjusting the camera speed and video length of the final video, and finally displays the navigation results.

[0013] S5: Construct a candidate generation module to generate rough navigation results through keywords and corresponding weights;

[0014] S6: Build a global discriminator and a local discriminator, combine the preset demonstration video and professional shooting knowledge, and continuously optimize its own prediction results during the training process, thereby improving the accuracy of panoramic video navigation results and user satisfaction;

[0015] S7: Repeat steps S3 and S4 to gradually optimize the performance and generalization ability of the model;

[0016] S8: Evaluate the performance and accuracy of the model using a test set, where the test set includes color video frames in ERP format and the corresponding optimal viewport center point coordinates.

[0017] Furthermore, the preset video large language model adopts the Video-LLaVA model.

[0018] Further, the step S2 specifically includes:

[0019] S201: Use the Video-LLaVA model to perform preliminary semantic analysis on the videos in the training set;

[0020] S202: extracting high-level semantic information of the entire video according to the semantic analysis result, the high-level semantic information includes the complete content of the video, and generating a text description according to the extracted high-level semantic information for subsequent keyword extraction.

[0021] Furthermore, the fine-tuned language model adopts a BERT model, and step S3 specifically includes:

[0022] S301: Obtain the original BERT model and fine-tune it according to the required keyword extraction task;

[0023] S302: Use the fine-tuned BERT model to extract key information from the text description output in step S2, and the extracted key information enters the next stage as candidate keywords.

[0024] Further, the step S4 specifically includes:

[0025] S401: Design a graphical user interface that allows users to upload local videos and displays the keywords extracted in step S3 above;

[0026] S402: The user selects a keyword that meets the preference by clicking, and then generates a console, and adjusts the preference level of each selected keyword through the console;

[0027] S403: The graphical user interface also provides a playback control function, and the user can adjust the final video result according to his / her own wishes;

[0028] S44: Display the final panoramic video navigation result, and the video played is a video with a normal viewing angle.

[0029] Further, the step S5 specifically includes:

[0030] S501: Design a candidate generation module, use the pre-trained Mask R-CNN model to segment the human elements in the panoramic video, and generate an accurate segmentation mask for each human instance;

[0031] S502: Design a candidate generation module, use the pre-trained Deeplab v3+ model to segment other key elements in the panoramic video, and generate accurate segmentation masks for each key element;

[0032] S503: After the semantic information is extracted, candidate normal view videos are generated as rough navigation results according to the weights of the keywords selected by the user and the playback control requirements.

[0033] Further, the step S6 specifically includes:

[0034] S601: Design a local discriminator to calculate the similarity between adjacent video frames to ensure the continuity and smooth transition of video segments;

[0035] The local discriminator is implemented through a two-branch fusion convolutional network with ResNet18 as the backbone. The local discriminator processes a single shot separately, processes the semantic focus of people and other key elements separately, and outputs In addition to the RGB image, the local discriminator also requires a mask image to better capture the composition of the corresponding semantics;

[0036] S602: Design a global discriminator, which is a deep neural network architecture, including multiple convolutional layers, pooling layers and several fully connected layers, and output a score, which is used to measure the overall quality of the entire video sequence;

[0037] The global discriminator is implemented by a sequence discriminator with ResNet18 as the backbone and Bi-LSTM as the classification head. The global discriminator jointly considers a sequence of w shots to investigate multiple shot combinations and outputs Each shot is represented by five keyframes;

[0038] S603: Combining local and global scoring criteria, the system searches for the best candidate video sequence through the BeamSearch strategy and outputs the final video;

[0039] The system incorporates professional shooting knowledge into the generated video to ensure the visual continuity and beauty between consecutive shots. If the following conditions are met: the movement of the frame is consistent with the movement of the character, and the frame movement direction of consecutive shots is consistent; the generated video score is The goal of this step is to retrieve the best candidate shots from each shot pool to form a video that is similar to the real data mode and follows the shooting rules. The local score aims to maximize the local score S loc , which only depends on two adjacent lenses, the formula is as follows:

[0040]

[0041] in, It reflects the semantic information of the jth video frame. If it meets the corresponding category, the value is 1, otherwise it is 0. The BeamSearch strategy is used to maximize the possible output navigation results S loc Fraction;

[0042] The global criterion ranks all samples to obtain the highest samples for selection; the global criterion jointly considers the score from the global discriminator and the shooting rules of the entire shooting sequence to form the following score:

[0043]

[0044] Among them, S g It is calculated in a sliding window of length w with a stride of w / 2, and the final output is the highest S among all samples. glo of the sample.

[0045] Further, the step S8 specifically includes:

[0046] S801: After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include: content coverage, time cost, and user satisfaction.

[0047] S802: According to the results of the performance evaluation, adjust the hyperparameters of the model, including the learning rate and the segmentation threshold, and optimize the candidate generation and video synthesis to improve the evaluation performance of the model.

[0048] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned panoramic video navigation methods driven by user subjective preferences is implemented.

[0049] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned panoramic video navigation methods driven by user subjective preferences.

[0050] Compared with the prior art, the advantages of the present invention are:

[0051] When facing panoramic videos with rich content, the navigation results predicted by existing methods often fail to meet the subjective wishes of users. The present invention uses a pre-trained large language model to extract key content from the entire video and presents these contents to users in the form of keywords. Users can customize navigation results according to their own preferences, so that the generated navigation path is closer to the user's viewing intention. This method not only improves user satisfaction, but also gives users greater control, making the viewing experience more personalized and interactive;

[0052] Traditional panoramic video navigation methods based on saliency only focus on low-level local information of the video and lack a deep understanding of the overall content of the video. Therefore, they often cannot fully reflect the key content of the video. The present invention overcomes this shortcoming by combining local semantic information and global semantic information. Global semantic information focuses on the overall content of the video, while local semantic information emphasizes the connection and transition of adjacent video clips. This combination of dual semantic information greatly improves the accuracy and quality of navigation results. In addition, the present invention also combines external knowledge learned from demonstration videos and professional shooting rules to fine-tune the navigation results, making them not only technically accurate, but also in line with video aesthetic standards, providing users with a better viewing experience;

[0053] The present invention designs an interactive and friendly graphical user interface, which makes the display of key information and the collection of user needs intuitive and convenient. Through this interface, users can easily view the key content of the video and quickly customize navigation parameters. The interface design focuses on user experience, making the operation simple and intuitive, and easy to use even for non-technical users. This design not only improves the user's efficiency, but also enhances the user's sense of participation and interaction when watching panoramic videos.

[0054] In summary, compared with the prior art, the present invention can more effectively handle panoramic video navigation tasks, has higher practical value and good market application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The framework diagram of the panoramic video navigation method driven by user subjective preference is shown;

[0056] Figure 2 is a schematic diagram of a graphical user interface;

[0057] Figure 3 Schematic diagram of the local discriminator structure;

[0058] Figure 4 Schematic diagram of the global discriminator structure;

[0059] Figure 5 This is a user evaluation comparison result of the panoramic video navigation method and some existing SOTA models. DETAILED DESCRIPTION

[0060] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0061] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0062] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0063] Embodiment 1:

[0064] Attached Figure 1 The flowchart of some embodiments of the panoramic video navigation method driven by user subjective preference according to the present disclosure is shown, which includes the following steps:

[0065] S1: Obtain several panoramic videos in ERP format and preprocess them to form training sets and test sets.

[0066] First, video frames are extracted from the panoramic video in ERP format to obtain panoramic video frame images in ERP format. Preferably, in order to increase the diversity of video content and improve the robustness of the model, S1 includes: selecting videos with rich motion types and semantic information from the panoramic videos to be selected to form a training set and a test set.

[0067] The selection of motion type richness adopts optical flow analysis technology, uses optical flow algorithm to detect motion information in the video, and analyzes the motion of pixels between frames. The selection of semantic information richness adopts semantic segmentation technology to classify video frames at pixel level to obtain detailed scene information.

[0068] Further, in a specific embodiment, the present invention uses the existing image segmentation and semantic understanding task datasets Cityscapes and Ade20K for training and testing. The Cityscapes dataset is a dataset that focuses on semantic understanding tasks in urban environments. It is mainly used to evaluate and train semantic segmentation, instance segmentation, and object detection models. The dataset consists of 5,000 high-resolution images from 50 different cities, of which 2,975 are used for training, 500 are used for verification, and 1,525 are used for testing. Each image provides detailed pixel-level annotations, including categories such as roads, buildings, vehicles, pedestrians, etc. The Ade20K dataset is a large-scale, multi-scene semantic segmentation dataset that covers a wider range of daily scenes and object categories. The dataset contains 20,000 images for training, 2,000 for verification, and 3,000 for testing. These images cover a variety of indoor and outdoor scenes, including residences, offices, natural landscapes, city streets, etc. Since the Ade20K dataset is relatively large, the present invention randomly extracts 2000 images for training, 200 for verification, and 300 for testing.

[0069] S2: Use the existing trained video language model to perform preliminary semantic analysis on the images in the training set to extract high-level semantic information of the entire video and generate text descriptions based on the extracted semantic information.

[0070] In a specific implementation, the existing trained video large language model uses a Video-LLaVA model.

[0071] S21: Use the existing trained video language model to perform preliminary semantic analysis on the videos in the training set.

[0072] In a more specific implementation, the preliminary semantic parsing method is: inputting the color panoramic video in ERP format into an existing trained video large language model Video-LLaVA to obtain a preliminary semantic parsing result.

[0073] S22: extracting high-level semantic information of the entire video based on the semantic analysis result. The high-level semantic information summarizes the complete content of the video. Generate a text description based on the extracted high-level semantic information for subsequent keyword extraction.

[0074] In the Video-LLaVA workflow, Video-LLaVA uses the LanguageBind encoder to extract features from the original visual signal. The LanguageBind encoder can map different modalities into the text feature space, providing a unified visual representation. Subsequently, OpenCLIP is initialized to naturally align images and languages ​​in a shared feature space. By using 3 million video-text pairs to align video representations into the language space, the visual and language signals are combined to generate accurate text subtitles.

[0075] S3: Use the fine-tuned language model to extract key semantic information from the text description of the video as the subsequent keywords to be selected.

[0076] In a specific implementation, the fine-tuned language model is based on a BERT model.

[0077] S31: Get the original BERT model and fine-tune it according to the required keyword extraction task.

[0078] In a more specific embodiment, the fine-tuning operation is as follows: first, collect and annotate text data containing keywords, and convert the text into an input format acceptable to the BERT model. The keyword extraction task is designed as a sequence labeling task, with each word corresponding to a label. Then, a fully connected layer and a softmax activation function are added on top of the output of the BERT model to perform sequence labeling. Use the annotated data for training, adjust the BERT model parameters, and optimize the keyword extraction performance. Finally, the cross entropy loss function is used to adjust the model parameters by minimizing the loss function to optimize the keyword extraction performance.

[0079] S32: Use the fine-tuned BERT model to extract key information from the text description output in the previous step, and the extracted key information will enter the next stage as candidate keywords.

[0080] In a specific implementation, the keywords represent key information of the entire video and are presented as user options in a graphical user interface.

[0081] S4: Design a graphical user interface that allows users to upload local videos and display the extracted keywords. Users select several keywords and set weights, which are then sent to the deep neural network model for processing. It also provides playback control functions, such as adjusting the camera speed and video length of the final video, and finally displays the navigation results.

[0082] In a specific embodiment, the graphical user interface is developed using PyQt5, which is a powerful Python library that allows desktop applications to be written in Python and provides a rich set of GUI components. Its powerful layout management can facilitate complex interface layout design. The developed graphical user interface is shown in the attached figure. Figure 2 shown.

[0083] S41: Design a graphical user interface that allows users to upload local videos and displays the keywords extracted in the previous step.

[0084] In a specific embodiment, the graphical user interface allows the user to click a folder icon to browse local files, select a local video to send to the system for keyword extraction, and display the extracted keywords on the interface.

[0085] S42: The user selects a keyword that meets the user's preference by clicking, and then generates a console, through which the user adjusts the preference level of each selected keyword.

[0086] In a specific embodiment, the graphical user interface allows the user to select a number of keywords, and then generates a console in the interface to control the weight of each keyword. The console is presented in the form of a slider, and the user adjusts the weight value by dragging the slider. If the user does not select a keyword, the system will not combine the keyword information.

[0087] S43: The graphical user interface also provides playback control functions, allowing users to adjust the final video result according to their own wishes.

[0088] In one embodiment, the graphical user interface allows the user to adjust the camera speed and the length of the generated video, also in the form of slider controls.

[0089] S44: Display the final panoramic video navigation result, and the video played is a video with a normal viewing angle.

[0090] In a specific embodiment, the graphical user interface displays the navigation results on the right side, as shown in the attached figure. Figure 2 As shown, similar to a general video player, it provides video play / pause and volume adjustment functions.

[0091] S5: Design a candidate generation module, which generates rough navigation results through keywords and corresponding weights. The semantic information of the rough navigation results is as consistent as possible with the keywords selected by the user.

[0092] In a specific implementation, the candidate generation module preprocesses the panoramic video, extracts semantic information, generates and scores candidate normal view videos, and provides high-quality candidate segments for the video synthesis module. By using the Mask R-CNN and Deeplabv3+ models to segment people and buildings, combined with the semantic content and shooting style specified by the user, it ensures that the generated video meets the user's needs and improves the quality and usability of the final video.

[0093] S51: Design a candidate generation module to segment people in panoramic videos using the pre-trained Mask R-CNN model and generate accurate segmentation masks for each human instance.

[0094] In a specific embodiment, the MaskR-CNN model is based on the FasterR-CNN architecture and adds a segmentation branch, which can perform object detection and instance segmentation at the same time, and is suitable for dealing with complex object shapes and occlusion problems. It uses a region proposal network to generate candidate regions, and then classifies and segments each region, which is very suitable for dealing with highly diverse and complex objects such as people. The MaskR-CNN model is pre-trained on the Cityscapes dataset.

[0095] S52: Design a candidate generation module to segment other key elements (such as buildings, vehicles) in panoramic videos using the pre-trained Deeplabv3+ model and generate accurate segmentation masks for each key element.

[0096] In a specific embodiment, the Deeplabv3+ model is a semantic segmentation network that uses dilated convolution and spatial pyramid pooling techniques to effectively capture features of different scales and is suitable for processing other key elements besides humans, such as buildings and vehicles, which have strong regularity and clear boundaries. The Deeplabv3+ model is pre-trained on the Ade20K dataset.

[0097] S53: After the semantic information is extracted, candidate normal view videos are generated as rough navigation results according to the weights of the keywords selected by the user and the playback control requirements.

[0098] In a specific implementation, the system extracts information such as the category, location, and size of the target from the segmentation mask generated by the MaskR-CNN and Deeplabv3+ models. The score of each segment is calculated based on the weight of the user's keywords and sorted by score. The sorted segments are filtered and adjusted according to the user's requirements such as the playback speed and perspective switching frequency. The filtered segments are spliced ​​into a complete video as a rough navigation result.

[0099] S6: Design a global discriminator and a local discriminator, combine demonstration videos and professional shooting knowledge, and continuously optimize their own prediction results during the training process, thereby improving the accuracy of panoramic video navigation results and user satisfaction.

[0100] In a specific embodiment, the global discriminator and the local discriminator are data-driven and are intended to evaluate whether the generated shots follow the actual data patterns in terms of instance elements and shot composition. Figure 3 The network architecture of the local discriminator of the present invention is shown in FIG. Figure 4 The network architecture of the global discriminator of the present invention is shown.

[0101] S61: Design a local discriminator to calculate the similarity between adjacent video frames to ensure the continuity and smooth transition of video clips.

[0102] In a specific embodiment, the local discriminator is implemented by a dual-branch fusion convolutional network with ResNet18 as the backbone. The local discriminator processes a single shot separately, processes the semantic focus of people and other key elements (such as buildings, vehicles), and outputs Etc. In addition to the RGB image, the local discriminator also requires a mask image to better capture the composition of the corresponding semantics.

[0103] S62: Design a global discriminator, which is a deep neural network architecture consisting of multiple convolutional layers, pooling layers, and several fully connected layers, and outputs a score that is used to measure the overall quality of the entire video sequence.

[0104] In a specific embodiment, the global discriminator is implemented by a sequence discriminator, which uses ResNet18 as the backbone and Bi-LSTM as the classification head. The global discriminator considers a series of w shots together to investigate multiple shot combinations and outputs Each shot is represented by five keyframes.

[0105] S63: Combining local and global scoring criteria, the system searches for the best candidate video sequence through the BeamSearch strategy and outputs the final video.

[0106] In a specific embodiment, the system incorporates professional shooting knowledge into the generated video to ensure visual continuity and beauty between consecutive shots. If: 1) the movement of the frame is consistent with the movement of the character;

[0107] 2) If the frames of consecutive shots move in the same direction, the generated video score is The goal of this step is to retrieve the best candidate shots from each shot pool to compose a video that is similar to the real data mode and follows the shooting rules. The local score aims to maximize the local score S loc , which only depends on two adjacent lenses, the formula is as follows:

[0108]

[0109] in, It reflects the semantic information of the jth video frame. If it meets the corresponding category, the value is 1, otherwise it is 0. The BeamSearch strategy is used to maximize the possible output navigation results S loc Fraction.

[0110] The global criterion ranks all samples to obtain the highest samples for selection. The global criterion jointly considers the score from the global discriminator and the shooting rules of the entire shooting sequence to form the following score:

[0111]

[0112] Among them, S g It is calculated in a sliding window of length w with a stride of w / 2. The final output is the highest S among all samples. glo of the sample.

[0113] S7: Repeat the above steps S3 and S4 several times to gradually optimize the performance and generalization ability of the model.

[0114] In some specific implementations, the more repetitions are made, the closer the generated navigation results are to the real data, and the smaller the performance improvement is. When the performance is no longer improved, the training is terminated.

[0115] S8: Evaluate the performance and accuracy of the model using an independent validation set or test set, where the validation set or test set contains color video frames in ERP format and the corresponding optimal viewport center point coordinates.

[0116] After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include but are not limited to content coverage, time cost, and user satisfaction.

[0117] According to the results of the performance evaluation, the model's hyperparameters, such as learning rate, segmentation threshold, etc., are adjusted to optimize candidate generation and video synthesis to improve the model's evaluation performance.

[0118] The graphical user interface, candidate generation module, global discriminator, local discriminator and video fusion strategy in the above process together constitute a panoramic video navigation method driven by user subjective preferences.

[0119] In a specific embodiment, according to the above specific implementation, the following can be obtained: Figure 5 The user evaluation comparison result diagram of the panoramic video navigation method driven by user subjective preferences of the present invention and some existing SOTA models is shown, based on the following three evaluation annotations: 1) Which method is easier to locate important events? 2) Which method is more in line with personal preferences? 3) Which method do you prefer? The results show that none of the compared existing model methods can generate navigation results that meet the user's personal preferences. A small number of users prefer other methods because the method of the present invention requires users to manually select preferred content before generating navigation results, which adds an extra burden to users who do not care much about this requirement. Overall, the present invention still meets the needs of most users, and proves the effectiveness of combining video semantic information with user-selected keyword semantics in the field of panoramic video navigation.

[0120] Embodiment 2:

[0121] This embodiment provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a panoramic video navigation method driven by user subjective preferences, including the following steps:

[0122] S1: Obtain multiple panoramic videos in ERP format and perform preprocessing to obtain training sets and test sets;

[0123] S2: Use the preset video language model to perform preliminary semantic analysis on the images in the training set, extract high-level semantic information of the entire video, and generate text descriptions based on the extracted semantic information;

[0124] S3: Use the fine-tuned language model to extract key semantic information from the text description of the video as the keywords to be selected later;

[0125] S4: Build a graphical user interface that allows users to upload local videos and display the extracted keywords. Users select multiple keywords and set weights, which are then fed into the deep neural network model for processing. It also provides playback control functions, including adjusting the camera speed and video length of the final video, and finally displays the navigation results.

[0126] S5: Construct a candidate generation module to generate rough navigation results through keywords and corresponding weights;

[0127] S6: Build a global discriminator and a local discriminator, combine the preset demonstration video and professional shooting knowledge, and continuously optimize its own prediction results during the training process, thereby improving the accuracy of panoramic video navigation results and user satisfaction;

[0128] S7: Repeat steps S3 and S4 to gradually optimize the performance and generalization ability of the model;

[0129] S8: Evaluate the performance and accuracy of the model using a test set, where the test set includes color video frames in ERP format and the corresponding optimal viewport center point coordinates.

[0130] Embodiment 3:

[0131] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0132] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the panoramic video navigation method driven by the user's subjective preference in the above embodiment; the processor may load and execute the following steps:

[0133] S1: Obtain multiple panoramic videos in ERP format and perform preprocessing to obtain training sets and test sets;

[0134] S2: Use the preset video language model to perform preliminary semantic analysis on the images in the training set, extract high-level semantic information of the entire video, and generate text descriptions based on the extracted semantic information;

[0135] S3: Use the fine-tuned language model to extract key semantic information from the text description of the video as the keywords to be selected later;

[0136] S4: Build a graphical user interface that allows users to upload local videos and display the extracted keywords. Users select multiple keywords and set weights, which are then fed into the deep neural network model for processing. It also provides playback control functions, including adjusting the camera speed and video length of the final video, and finally displays the navigation results.

[0137] S5: Construct a candidate generation module to generate rough navigation results through keywords and corresponding weights;

[0138] S6: Build a global discriminator and a local discriminator, combine the preset demonstration video and professional shooting knowledge, and continuously optimize its own prediction results during the training process, thereby improving the accuracy of panoramic video navigation results and user satisfaction;

[0139] S7: Repeat steps S3 and S4 to gradually optimize the performance and generalization ability of the model;

[0140] S8: Evaluate the performance and accuracy of the model using a test set, where the test set includes color video frames in ERP format and the corresponding optimal viewport center point coordinates.

[0141] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0146] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A panoramic video navigation method driven by user subjective preference, characterized in that: The method comprises: S1: Obtain multiple panoramic videos in ERP format and perform preprocessing to obtain training sets and test sets; S2: Use the preset video language model to perform preliminary semantic analysis on the images in the training set, extract high-level semantic information of the entire video, and generate text descriptions based on the extracted semantic information; S3: Use the fine-tuned language model to extract key semantic information from the text description of the video as the keywords to be selected later; S4: Build a graphical user interface that allows users to upload local videos and display the extracted keywords. Users select multiple keywords and set weights, which are then fed into the deep neural network model for processing. It also provides playback control functions, including adjusting the camera speed and video length of the final video, and finally displays the navigation results. S5: Construct a candidate generation module to generate rough navigation results through keywords and corresponding weights; S6: Build a global discriminator and a local discriminator, combine the preset demonstration video and professional shooting knowledge, and continuously optimize its own prediction results during the training process, so as to improve the accuracy of panoramic video navigation results and user satisfaction; S7: Repeat steps S3 and S4 to gradually optimize the performance and generalization ability of the model; S8: Evaluate the performance and accuracy of the model using a test set, where the test set includes color video frames in ERP format and the corresponding optimal viewport center point coordinates.

2. The method for panoramic video navigation driven by user subjective preference according to claim 1, characterized in that: The preset video large language model adopts the Video-LLaVA model.

3. The method for panoramic video navigation driven by user subjective preference according to claim 2, characterized in that: The step S2 specifically includes: S201: Use the Video-LLaVA model to perform preliminary semantic analysis on the videos in the training set; S202: extracting high-level semantic information of the entire video according to the semantic analysis result, the high-level semantic information includes the complete content of the video, and generating a text description according to the extracted high-level semantic information for subsequent keyword extraction.

4. The method for panoramic video navigation driven by user subjective preference according to claim 1, characterized in that: The fine-tuned language model adopts the BERT model, and step S3 specifically includes: S301: Obtain the original BERT model and fine-tune it according to the required keyword extraction task; S302: Use the fine-tuned BERT model to extract key information from the text description output in step S2, and the extracted key information enters the next stage as candidate keywords.

5. The method for panoramic video navigation driven by user subjective preference according to claim 1, characterized in that: The step S4 specifically includes: S401: Design a graphical user interface that allows users to upload local videos and displays the keywords extracted in step S3 above; S402: The user selects a keyword that meets the preference by clicking, and then generates a console, and adjusts the preference level of each selected keyword through the console; S403: The graphical user interface also provides a playback control function, and the user can adjust the final video result according to his / her own wishes; S44: Display the final panoramic video navigation result, and the video played is a video with a normal viewing angle.

6. The method for panoramic video navigation driven by user subjective preference according to claim 1, characterized in that: The step S5 specifically includes: S501: Design a candidate generation module, use the pre-trained Mask R-CNN model to segment the human elements in the panoramic video, and generate an accurate segmentation mask for each human instance; S502: Design a candidate generation module, use the pre-trained Deeplab v3+ model to segment other key elements in the panoramic video, and generate accurate segmentation masks for each key element; S503: After the semantic information is extracted, candidate normal view videos are generated as rough navigation results according to the weights of the keywords selected by the user and the playback control requirements.

7. The method for panoramic video navigation driven by user subjective preference according to claim 1, characterized in that: The step S6 specifically includes: S601: Design a local discriminator to calculate the similarity between adjacent video frames to ensure the continuity and smooth transition of video segments; The local discriminator is implemented through a two-branch fusion convolutional network with ResNet18 as the backbone. The local discriminator processes a single shot separately, processes the semantic focus of people and other key elements separately, and outputs In addition to the RGB image, the local discriminator also requires a mask image to better capture the composition of the corresponding semantics; S602: Design a global discriminator, which is a deep neural network architecture, including multiple convolutional layers, pooling layers and several fully connected layers, and output a score, which is used to measure the overall quality of the entire video sequence; The global discriminator is implemented by a sequence discriminator with ResNet18 as the backbone and Bi-LSTM as the classification head. The global discriminator jointly considers a sequence of w shots to investigate multiple shot combinations and outputs Each shot is represented by five keyframes; S603: Combining local and global scoring criteria, the system searches for the best candidate video sequence through the BeamSearch strategy and outputs the final video; The system incorporates professional shooting knowledge into the generated video to ensure the visual continuity and beauty between consecutive shots. If the following conditions are met: the movement of the frame is consistent with the movement of the character, and the frame movement direction of consecutive shots is consistent; the generated video score is The goal of this step is to retrieve the best candidate shots from each shot pool to form a video that is similar to the real data mode and follows the shooting rules. The local score aims to maximize the local score S loc , which only depends on two adjacent lenses, the formula is as follows: in, It reflects the semantic information of the jth video frame. If it meets the corresponding category, the value is 1, otherwise it is 0. The BeamSearch strategy is used to maximize the possible output navigation results S loc Fraction; The global criterion ranks all samples to obtain the highest samples for selection; the global criterion jointly considers the score from the global discriminator and the shooting rules of the entire shooting sequence to form the following score: Among them, S g It is calculated in a sliding window of length w with a stride of w / 2, and the final output is the highest S among all samples. glo of the sample.

8. The method for panoramic video navigation driven by user subjective preference according to claim 1, characterized in that: The step S8 specifically includes: S801: After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include: content coverage, time cost, and user satisfaction. S802: According to the results of the performance evaluation, adjust the hyperparameters of the model, including the learning rate and the segmentation threshold, and optimize the candidate generation and video synthesis to improve the evaluation performance of the model.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a panoramic video navigation method driven by user subjective preference as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a panoramic video navigation method driven by user subjective preferences as described in any one of claims 1 to 8.

Images