Immersive video quality evaluation method and device based on six-degree-of-freedom information guidance

Abstract

The invention discloses an immersive video quality evaluation method and device based on six-degree-of-freedom information guidance, and relates to the field of computer vision, and the method comprises the steps: constructing and training an immersive video quality evaluation model based on six-degree-of-freedom information guidance, and obtaining a trained evaluation model; extracting texture video blocks, texture key frames and depth key frames from the immersive video; inputting the texture video blocks, the texture key frames and the depth key frames into a trained evaluation model, and extracting corresponding visual features through a visual information coding module; inputting the visual features into a space-time mapping module to obtain a time visual mark and a space visual mark; encoding the instruction information and the six-degree-of-freedom viewpoint position information through a language encoder to obtain a text instruction mark and a viewpoint position mark; and combining the marks to obtain a combined mark, and inputting the combined mark into a voice decoder to obtain an immersive video quality score. According to the invention, the quality of the immersive video can be accurately and objectively evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and video evaluation, and in particular to an immersive video quality evaluation method and device based on six-degree-of-freedom information guidance. Background Art

[0002] With the rapid development of digital multimedia technology, the concept of the metaverse has been deeply rooted in people's minds and has spawned a series of new applications that integrate virtual and reality technologies, among which immersive video is particularly eye-catching. Immersive video technology has demonstrated its wide application potential in many fields such as distance education and medical care. Compared with traditional videos, immersive videos not only contain richer texture details, but also incorporate complex parallax information, bringing users an unprecedented visual experience.

[0003] However, in the process of immersive video acquisition, transmission and display, it is inevitable to encounter distortion problems such as motion blur and Gaussian noise. These problems seriously affect the quality of the video and user experience. Therefore, developing an algorithm that can accurately evaluate the quality of immersive video and conform to human visual perception not only has far-reaching research significance, but also indicates broad application prospects, which will provide a solid foundation for the further development and optimization of immersive video technology. Summary of the invention

[0004] The purpose of the present invention is to provide an immersive video quality evaluation method and device based on six-degree-of-freedom information guidance, which can accurately evaluate the immersive video quality that conforms to human visual perception.

[0005] The present invention adopts the following technical solution:

[0006] In a first aspect, an immersive video quality evaluation method based on six-degree-of-freedom information guidance includes:

[0007] An immersive video quality assessment model guided by six degrees of freedom information is constructed and trained to obtain a trained immersive video quality assessment model; the immersive video quality assessment model includes a visual information encoding module, a spatiotemporal mapping module, a language encoder and a speech decoder of a large language model;

[0008] Acquire an immersive video including texture videos and depth videos from multiple viewpoints, extract a number of texture video blocks and texture key frames from the texture videos from multiple viewpoints, and extract a number of depth key frames from the depth videos from multiple viewpoints;

[0009] Several texture video blocks, texture key frames and depth key frames are input into the trained immersive video quality evaluation model, and the texture video blocks, texture key frames and depth key frames of multiple viewpoints are respectively extracted with the visual information encoding module to obtain the corresponding visual features; the visual features are input into the spatiotemporal mapping module to obtain temporal visual tags and spatial visual tags; the instruction information and six-degree-of-freedom viewpoint position information are encoded through the language encoder to obtain text instruction tags and viewpoint position tags; the temporal visual tags, spatial visual tags, text instruction tags and viewpoint position tags are combined to obtain combined tags, and the combined tags are input into the speech decoder to obtain the immersive video quality score.

[0010] Preferably, a plurality of texture video blocks and texture key frames are extracted from texture videos of multiple viewpoints, and a plurality of depth key frames are extracted from depth videos of multiple viewpoints, as follows:

[0011] Texture video for each viewpoint and Deep Video Divide the video into blocks to obtain K consecutive texture video blocks and K consecutive depth video blocks; wherein, represents the j-th texture video block, represents the jth depth video block, N represents the number of frames of the texture video or the depth video for each viewpoint, and the number of frames of the texture video or the depth video for each viewpoint is equal, Represents the first frame, Indicates the number of video frames contained in each texture video block or depth video block. ; For each texture video block and depth video block, take the first frame as the key frame of the block to get the texture key frame and depth keyframes .

[0012] Preferably, the texture video blocks, texture key frames and depth key frames of multiple viewpoints are subjected to feature extraction by a visual information encoding module to obtain different visual features, specifically including:

[0013] The deep visual encoder and spatial visual encoder of the visual information encoding module are used to encode the deep keyframes. and texture keyframes Perform feature extraction separately to obtain the deep key frame features of each viewpoint and texture keyframe features ; The texture video block features are extracted through the temporal visual encoder of the visual information encoding module to obtain the texture video block features of multiple viewpoints , as shown below:

[0014] ;

[0015] ;

[0016] ;

[0017] in, Represents feature splicing; , and They represent the deep visual encoder, spatial visual encoder, and temporal visual encoder, respectively; i represents the i-th viewpoint, H, W, and C represent the height, width, and number of channels of the feature, respectively. Represents the set of real numbers.

[0018] Preferably, the visual features are input into the spatiotemporal mapping module to obtain temporal visual labels and spatial visual labels, specifically including:

[0019] The depth key frame features and texture key frame features are input into the space mapping unit of the spatiotemporal mapping module to obtain spatial visual tags, and the texture video block features are input into the time mapping unit to obtain temporal visual tags, as shown in the following formula:

[0020] ;

[0021] ;

[0022] in, Represents a spatial mapping unit; represents a time mapping unit; Represents spatial visual markers; Indicates a visual marker of time.

[0023] Preferably, the instruction information and the six-degree-of-freedom viewpoint position information are encoded by a language encoder to obtain a text instruction mark and a viewpoint position mark, which specifically includes:

[0024] The instruction information is encoded by the language encoder; the instruction information includes system instructions, guide instructions and response restrictions, the system instructions are used to inform the large language model of its functions, the guide instructions are used to inform the large language model of its specific tasks, and the response restrictions are used to inform the large language model of the specific requirements for answering questions;

[0025] The text instruction tag obtained after encoding the instruction information is as shown in the following formula:

[0026] ;

[0027] in, Indicates instruction information; represents a language encoder; Represents a text instruction tag;

[0028] The six-degree-of-freedom viewpoint position information is encoded by the language encoder to obtain a viewpoint position mark, as shown in the following formula:

[0029] ;

[0030] in, Represents the position information of viewpoint i; Represents the position tag of viewpoint i.

[0031] Preferably, the immersive video quality score obtained is as follows:

[0032] ;

[0033] ;

[0034] in, Speech decoder representing a large language model; represents the obtained immersive video quality score; Represents the synthetic features of each viewpoint.

[0035] In a second aspect, an immersive video quality assessment device based on six-degree-of-freedom information guidance includes:

[0036] A model building and training module is configured to build and train an immersive video quality assessment model guided by six degrees of freedom information to obtain a trained immersive video quality assessment model; the immersive video quality assessment model includes a visual information encoding module, a spatiotemporal mapping module, a language encoder and a speech decoder of a large language model;

[0037] A data extraction module is configured to obtain an immersive video including texture videos and depth videos of multiple viewpoints, extract a number of texture video blocks and texture key frames from the texture videos of the multiple viewpoints, and extract a number of depth key frames from the depth videos of the multiple viewpoints;

[0038] The video quality assessment module is configured to input a number of texture video blocks, texture key frames and depth key frames into a trained immersive video quality assessment model, extract features of the texture video blocks, texture key frames and depth key frames of multiple viewpoints through a visual information encoding module, and obtain corresponding visual features; input the visual features into a spatiotemporal mapping module to obtain temporal visual tags and spatial visual tags; encode the instruction information and six-degree-of-freedom viewpoint position information through a language encoder to obtain text instruction tags and viewpoint position tags; combine the temporal visual tags, spatial visual tags, text instruction tags and viewpoint position tags to obtain combined tags, and input the combined tags into a speech decoder to obtain an immersive video quality score.

[0039] According to a third aspect, an electronic device includes:

[0040] one or more processors;

[0041] A storage device for storing one or more programs;

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the immersive video quality assessment methods guided by six-degree-of-freedom information.

[0043] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the immersive video quality assessment methods based on six-degree-of-freedom information guidance.

[0044] In a fifth aspect, a computer program product includes a computer program, which, when executed by a processor, implements any of the immersive video quality assessment methods based on six-degree-of-freedom information guidance.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The spatiotemporal mapping module in the immersive video quality evaluation model constructed by the present invention maps spatial visual features and temporal visual features to text space respectively, which can improve the perception ability of different visual features;

[0047] (2) The language encoder in the immersive video quality assessment model constructed by the present invention guides visual features by encoding six-degree-of-freedom viewpoint position information and text guidance instructions, thereby improving the cross-modal understanding ability of the large language model;

[0048] (3) The present invention integrates the quality perception markers of immersive videos from the perspective of visual information processing and text instruction guidance. The speech decoder of the large language model can effectively learn and process visual markers and instruction markers, thereby completing the quality evaluation of the immersive video. The evaluation results are highly consistent with the characteristics of human visual perception, providing a scientific and effective solution for the objective evaluation of immersive video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of an immersive video quality assessment method based on six-degree-of-freedom information guidance provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the structure of an immersive video quality evaluation model provided by an embodiment of the present invention;

[0051] Figure 3 A structural block diagram of an immersive video quality assessment device based on six-degree-of-freedom information guidance provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0054] See also Figure 1 As shown, this embodiment provides an immersive video quality assessment method based on six-degree-of-freedom information guidance, including the following steps.

[0055] S101, constructing and training an immersive video quality evaluation model guided by six-degree-of-freedom information to obtain a trained immersive video quality evaluation model.

[0056] For details, see Figure 2 As shown, it is a schematic diagram of the structure of the immersive video quality assessment model provided by an embodiment of the present invention, and the immersive video quality assessment model includes a visual information encoding module, a spatiotemporal mapping module, a language encoder and a speech decoder of a large language model. The visual information encoding module includes a deep visual encoder, a spatial visual encoder and a temporal visual encoder. The spatiotemporal mapping module includes a spatial mapping unit and a temporal mapping unit.

[0057] The proposed immersive video quality assessment model based on six-degree-of-freedom information guidance uses Pytorch to build the environment, and uses NVIDIA RTX A6000 GPU for experiments. The experimental dataset is IMVD, 80% of which is used for training. In the training stage, the input resolution is adjusted to 448×448 for visual encoding, and the corresponding text guidance information is fused. The model uses the pre-trained Vision Transformer as the spatial visual encoder, Resnet as the deep visual encoder, Slowfast as the temporal visual encoder, and the language codec uses the language codec of the large language model Llama. In the specific implementation, the parameters of the language codec and the visual encoder are fixed, and only the spatiotemporal mapping module is trained and adjusted to achieve modal alignment of visual text tags. S102, obtain an immersive video containing texture videos and depth videos from multiple viewpoints, extract a number of texture video blocks and texture key frames from the texture videos from multiple viewpoints, and extract a number of depth key frames from the depth videos from multiple viewpoints.

[0058] Specifically, the texture video and depth video of each viewpoint are divided into blocks, the length of each video block is consistent, and each viewpoint obtains K consecutive texture video blocks and K consecutive depth video blocks. For each texture video block and depth video block, the first frame in the block is taken as the key frame of the block, thereby obtaining several video blocks and key frames.

[0059] In this embodiment, the texture video of each viewpoint and deep video Divide the video into blocks to obtain K consecutive texture video blocks and K consecutive depth video blocks; wherein, represents the j-th texture video block, represents the jth depth video block, N represents the number of frames of the texture video or the depth video for each viewpoint, and the number of frames of the texture video or the depth video for each viewpoint is equal, Represents the first frame, Indicates the number of video frames contained in each texture video block or depth video block. ; For each texture video block and depth video block, take the first frame as the key frame of the block to get the texture key frame and depth keyframes .

[0060] S103, inputting a number of texture video blocks, texture key frames and depth key frames into a trained immersive video quality evaluation model, performing feature extraction on the texture video blocks, texture key frames and depth key frames of multiple viewpoints respectively through a visual information encoding module to obtain corresponding visual features; inputting the visual features into a spatiotemporal mapping module to obtain temporal visual tags and spatial visual tags; encoding the instruction information and the six-degree-of-freedom viewpoint position information through a language encoder to obtain text instruction tags and viewpoint position tags; combining the temporal visual tags, spatial visual tags, text instruction tags and viewpoint position tags to obtain a combined tag, and inputting the combined tag into a speech decoder to obtain an immersive video quality score.

[0061] In this embodiment, the texture video blocks, texture key frames and depth key frames of multiple viewpoints are feature extracted by the visual information encoding module to obtain different visual features, including:

[0062] The deep visual encoder and spatial visual encoder of the visual information encoding module are used to encode the deep keyframes. and texture keyframes Perform feature extraction separately to obtain the deep key frame features of each viewpoint and texture keyframe features ; The texture video block features are extracted through the temporal visual encoder of the visual information encoding module to obtain the texture video block features of multiple viewpoints , as shown below:

[0063] ;

[0064] ;

[0065] ;

[0066] in, Represents feature splicing; , and They represent the deep visual encoder, spatial visual encoder, and temporal visual encoder, respectively; i represents the i-th viewpoint, H, W, and C represent the height, width, and number of channels of the feature, respectively. Represents the set of real numbers.

[0067] The visual features are input into the spatiotemporal mapping module to obtain temporal visual tags and spatial visual tags, including:

[0068] The depth key frame features and texture key frame features are input into the space mapping unit of the spatiotemporal mapping module to obtain spatial visual tags, and the texture video block features are input into the time mapping unit to obtain temporal visual tags, as shown in the following formula:

[0069] ;

[0070] ;

[0071] in, Represents a spatial mapping unit; represents a time mapping unit; Represents spatial visual markers; Indicates a visual marker of time.

[0072] The command information and the six-degree-of-freedom viewpoint position information are encoded by a language encoder to obtain text command tags and viewpoint position tags, which specifically include:

[0073] The instruction information is encoded by the language encoder; the instruction information includes system instructions, guide instructions and response restrictions, the system instructions are used to inform the large language model of its functions, the guide instructions are used to inform the large language model of its specific tasks, and the response restrictions are used to inform the large language model of the specific requirements for answering questions;

[0074] The text instruction tag obtained after encoding the instruction information is as shown in the following formula:

[0075] ;

[0076] in, Indicates instruction information; represents a language encoder; Represents a text instruction tag;

[0077] The six-degree-of-freedom viewpoint position information is encoded by the language encoder to obtain a viewpoint position mark, as shown in the following formula:

[0078] ;

[0079] in, Represents the position information of viewpoint i; Represents the position tag of viewpoint i.

[0080] The temporal visual marker, the spatial visual marker, the text instruction marker and the viewpoint position marker are combined to obtain a combined marker, and the combined marker is input into the speech decoder to obtain an immersive video quality score as follows:

[0081] ;

[0082] ;

[0083] in, Speech decoder representing a large language model; represents the obtained immersive video quality score; Represents the synthetic features of each viewpoint.

[0084] See also Figure 3 As shown, the present invention also discloses an immersive video quality evaluation device based on six-degree-of-freedom information guidance, comprising:

[0085] The model building and training module 301 is configured to build and train an immersive video quality assessment model guided by six degrees of freedom information to obtain a trained immersive video quality assessment model; the immersive video quality assessment model includes a visual information encoding module, a spatiotemporal mapping module, a language encoder and a speech decoder of a large language model;

[0086] The data extraction module 302 is configured to obtain an immersive video including texture videos and depth videos of multiple viewpoints, extract a plurality of texture video blocks and texture key frames from the texture videos of the multiple viewpoints, and extract a plurality of depth key frames from the depth videos of the multiple viewpoints;

[0087] The video quality assessment module 303 is configured to input a number of texture video blocks, texture key frames and depth key frames into a trained immersive video quality assessment model, extract features of the texture video blocks, texture key frames and depth key frames of multiple viewpoints respectively through the visual information encoding module to obtain corresponding visual features; input the visual features into the space-time mapping module to obtain temporal visual tags and spatial visual tags; encode the instruction information and the six-degree-of-freedom viewpoint position information through the language encoder to obtain text instruction tags and viewpoint position tags; combine the temporal visual tags, spatial visual tags, text instruction tags and viewpoint position tags to obtain combined tags, and input the combined tags into the speech decoder to obtain an immersive video quality score.

[0088] The specific implementation of each module of an immersive video quality evaluation device based on six-degree-of-freedom information guidance is the same as the immersive video quality evaluation method based on six-degree-of-freedom information guidance, and this embodiment will not be repeated.

[0089] See also Figure 4 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. Figure 4 As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0090] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0091] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0092] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0093] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0094] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0095] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0096] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0097] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0098] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0099] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0100] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0101] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0102] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0103] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An immersive video quality evaluation method based on six-degree-of-freedom information guidance, characterized in that: include: An immersive video quality assessment model guided by six degrees of freedom information is constructed and trained to obtain a trained immersive video quality assessment model; the immersive video quality assessment model includes a visual information encoding module, a spatiotemporal mapping module, a language encoder and a speech decoder of a large language model; Acquire an immersive video including texture videos and depth videos from multiple viewpoints, extract a number of texture video blocks and texture key frames from the texture videos from multiple viewpoints, and extract a number of depth key frames from the depth videos from multiple viewpoints; Several texture video blocks, texture key frames and depth key frames are input into the trained immersive video quality evaluation model, and the texture video blocks, texture key frames and depth key frames of multiple viewpoints are respectively extracted with the visual information encoding module to obtain the corresponding visual features; the visual features are input into the spatiotemporal mapping module to obtain the temporal visual mark and the spatial visual mark; the instruction information and the six-degree-of-freedom viewpoint position information are encoded with the language encoder to obtain the text instruction mark and the viewpoint position mark; The temporal visual marker, the spatial visual marker, the text instruction marker and the viewpoint position marker are combined to obtain a combined marker, and the combined marker is input into a speech decoder to obtain an immersive video quality score.

2. The immersive video quality assessment method based on six-degree-of-freedom information guidance according to claim 1, characterized in that: A plurality of texture video blocks and texture key frames are extracted from texture videos of multiple viewpoints, and a plurality of depth key frames are extracted from depth videos of multiple viewpoints, as follows: Texture video for each viewpoint and Deep Video Divide the video into blocks to obtain K consecutive texture video blocks and K consecutive depth video blocks; wherein, represents the j-th texture video block, represents the jth depth video block, N represents the number of frames of the texture video or the depth video for each viewpoint, and the number of frames of the texture video or the depth video for each viewpoint is equal, Represents the first frame, Indicates the number of video frames contained in each texture video block or depth video block. ; For each texture video block and depth video block, take the first frame as the key frame of the block to get the texture key frame and depth keyframes .

3. The immersive video quality assessment method based on six-degree-of-freedom information guidance according to claim 2, characterized in that: The visual information encoding module extracts features from texture video blocks, texture key frames and depth key frames from multiple viewpoints to obtain different visual features, including: The deep visual encoder and spatial visual encoder of the visual information encoding module are used to encode the deep keyframes. and texture keyframes Perform feature extraction separately to obtain the deep key frame features of each viewpoint and texture keyframe features ; The texture video block features are extracted through the temporal visual encoder of the visual information encoding module to obtain the texture video block features of multiple viewpoints , as shown below: ； ； ； in, Represents feature splicing; , and Represent the deep visual encoder, spatial visual encoder and temporal visual encoder respectively; i represents the i-th viewpoint, H, W and C represent the height, width and number of channels of the feature respectively, Represents the set of real numbers.

4. The immersive video quality assessment method based on six-degree-of-freedom information guidance according to claim 3 is characterized in that: The visual features are input into the spatiotemporal mapping module to obtain temporal visual tags and spatial visual tags, including: The depth key frame features and texture key frame features are input into the space mapping unit of the spatiotemporal mapping module to obtain spatial visual tags, and the texture video block features are input into the time mapping unit to obtain temporal visual tags, as shown in the following formula: ； ； in, Represents a spatial mapping unit; represents a time mapping unit; Represents spatial visual markers; Indicates a visual marker of time.

5. The immersive video quality assessment method based on six-degree-of-freedom information guidance according to claim 4 is characterized in that: The command information and the six-degree-of-freedom viewpoint position information are encoded by a language encoder to obtain text command tags and viewpoint position tags, which specifically include: The instruction information is encoded by the language encoder; the instruction information includes system instructions, guide instructions and response restrictions, the system instructions are used to inform the large language model of its functions, the guide instructions are used to inform the large language model of its specific tasks, and the response restrictions are used to inform the large language model of the specific requirements for answering questions; The text instruction tag obtained after encoding the instruction information is as shown in the following formula: ； in, Indicates instruction information; represents a language encoder; Represents a text instruction tag; The six-degree-of-freedom viewpoint position information is encoded by the language encoder to obtain a viewpoint position mark, as shown in the following formula: ； in, Represents the position information of viewpoint i; Represents the position tag of viewpoint i.

6. The immersive video quality assessment method based on six-degree-of-freedom information guidance according to claim 5, characterized in that: The resulting immersive video quality score is as follows: ； ； in, Speech decoder representing a large language model; represents the obtained immersive video quality score; Represents the synthetic features of each viewpoint.

7. An immersive video quality assessment device based on six-degree-of-freedom information guidance, characterized in that: include: A model building and training module is configured to build and train an immersive video quality assessment model guided by six degrees of freedom information to obtain a trained immersive video quality assessment model; the immersive video quality assessment model includes a visual information encoding module, a spatiotemporal mapping module, a language encoder and a speech decoder of a large language model; A data extraction module is configured to obtain an immersive video including texture videos and depth videos of multiple viewpoints, extract a number of texture video blocks and texture key frames from the texture videos of the multiple viewpoints, and extract a number of depth key frames from the depth videos of the multiple viewpoints; The video quality assessment module is configured to input a plurality of texture video blocks, texture key frames and depth key frames into a trained immersive video quality assessment model, extract features of the texture video blocks, texture key frames and depth key frames of multiple viewpoints respectively through a visual information encoding module to obtain corresponding visual features; input the visual features into a spatiotemporal mapping module to obtain temporal visual tags and spatial visual tags; encode the instruction information and the six-degree-of-freedom viewpoint position information through a language encoder to obtain text instruction tags and viewpoint position tags; The temporal visual marker, the spatial visual marker, the text instruction marker and the viewpoint position marker are combined to obtain a combined marker, and the combined marker is input into a speech decoder to obtain an immersive video quality score.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images