Panoramic image quality evaluation method and device based on psychological memory mechanism

Abstract

The present invention relates to the field of panoramic image quality assessment, and specifically to a method and device for panoramic image quality assessment based on a psychological memory mechanism. The method comprises obtaining textual semantic features from a distorted target panoramic image; obtaining multi-level image semantic features from the distorted target panoramic image; obtaining high-level semantic features from the textual semantic features and the multi-level image semantic features; performing viewport processing on the distorted target panoramic image to obtain a plurality of viewport images; obtaining local low-level semantic features for each viewport using each viewport image and the textual semantic features, and then concatenating the local low-level semantic features of all viewports to obtain low-level semantic features; and performing feature aggregation on the high-level semantic features and the low-level semantic features to obtain a quality assessment score for the distorted target panoramic image. In this way, visual and textual semantic information can be obtained with the help of a large model, and combined with the human brain's memory mechanism, the overall performance of the model can be effectively improved.

Description

Technical Field

[0001] The present invention generally relates to the field of panoramic image quality assessment, and more particularly, to a panoramic image quality assessment method and apparatus based on a psychological memory mechanism. Background Art

[0002] With the rapid development of camera technology, panoramic images are increasingly being used in entertainment, education, real estate, and other fields. Unlike traditional two-dimensional images, panoramic images offer high resolution, wide viewing angles, and spherical projection, enhancing user experience and comfort. However, panoramic images can introduce quality loss during acquisition, compression, transmission, and rendering. Therefore, effective quality assessment methods are needed to guide optimization and improve perceived image quality.

[0003] Traditional image quality assessment methods (such as PSNR and SSIM) are primarily targeted at planar images and cannot be directly applied to panoramic images. Research on panoramic image quality assessment methods has gradually attracted attention from both academia and industry. In the early stages of this field, researchers primarily manually extracted geometric and texture features from panoramic images and combined them with traditional quality assessment metrics for evaluation, resulting in mediocre performance. Subsequently, researchers have proposed a number of deep learning-based methods, such as using convolutional neural networks (CNNs) or visual transformers to extract features and predict image quality, which have improved performance. Considering that people can only view a single viewport at a time when viewing panoramic images, researchers have combined the characteristics of human vision to propose a panoramic image quality assessment method based on viewport features, further improving performance.

[0004] However, existing panoramic image quality evaluation methods have shortcomings in visual perception modeling and algorithm innovation. Summary of the Invention

[0005] According to an embodiment of the present invention, a panoramic image quality assessment solution based on psychological memory mechanisms is provided. This solution can use a large model to obtain visual and textual semantic information, and combined with the human brain's memory mechanism, it can effectively improve the overall performance of the model.

[0006] In a first aspect of the present invention, a method for evaluating panoramic image quality based on a psychological memory mechanism is provided. The method comprises:

[0007] Obtain a distorted panoramic image of the target;

[0008] Inputting the target distorted panoramic image into a text semantic feature extraction model to obtain text semantic features;

[0009] Inputting the target distorted panoramic image into a global visual semantic feature extraction model to obtain multi-level image semantic features; then inputting the text semantic features and the multi-level image semantic features into a global semantic interaction fusion model to obtain high-level semantic features;

[0010] The target distorted panoramic image is subjected to viewport processing to obtain a plurality of viewport images; each viewport image and the text semantic features are input into a local semantic interaction fusion model respectively, and a local low-level semantic feature of each viewport is output; and the local low-level semantic features of all viewports are then spliced to obtain a low-level semantic feature;

[0011] The high-level semantic features and the low-level semantic features are aggregated to obtain a quality evaluation score of the target distorted panoramic image.

[0012] In a second aspect of the present invention, a panoramic image quality assessment device based on a psychological memory mechanism is provided. The device comprises:

[0013] An acquisition module, used to obtain a distorted panoramic image of the target;

[0014] A first feature extraction module is configured to input the target distorted panoramic image into a text semantic feature extraction model to obtain text semantic features;

[0015] A second feature extraction module is configured to input the target distorted panorama into a global visual semantic feature extraction model to obtain multi-level image semantic features; and then input the text semantic features and the multi-level image semantic features into a global semantic interaction fusion model to obtain high-level semantic features;

[0016] A third feature extraction module is configured to perform viewport processing on the target distorted panoramic image to obtain a plurality of viewport images; input each viewport image and the text semantic features into a local semantic interaction fusion model, output a local low-level semantic feature of each viewport, and then splice the local low-level semantic features of all viewports to obtain a low-level semantic feature;

[0017] An evaluation module is used to perform feature aggregation on the high-level semantic features and the low-level semantic features to obtain a quality evaluation score of the target distorted panoramic image.

[0018] In a third aspect of the present invention, an electronic device is provided. The electronic device comprises at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the first aspect of the present invention.

[0019] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the first aspect of the present invention.

[0020] Compared with the prior art, the present invention has the following beneficial technical effects:

[0021] The multi-level interactive fusion of text information and visual information based on the human brain's memory mechanism can deeply explore the interactive relationship between the two modalities, extract rich semantic features, and improve the model's feature representation capabilities.

[0022] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0024] Figure 1 A flowchart of a panoramic image quality assessment method based on a psychological memory mechanism according to an embodiment of the present invention is shown;

[0025] Figure 2 A flowchart of a text semantic feature extraction model according to an embodiment of the present invention is shown;

[0026] Figure 3 A flowchart of a global semantic interaction fusion model according to an embodiment of the present invention is shown;

[0027] Figure 4 A flowchart of a semantic interaction fusion module according to an embodiment of the present invention is shown;

[0028] Figure 5 A flowchart of feature aggregation according to an embodiment of the present invention is shown;

[0029] Figure 6 A schematic diagram of a global visual semantic feature extraction model according to an embodiment of the present invention is shown;

[0030] Figure 7 A schematic diagram of a visual state space block according to an embodiment of the present invention is shown;

[0031] Figure 8 A schematic diagram of a global semantic interaction fusion model according to an embodiment of the present invention is shown;

[0032] Figure 9A schematic diagram of a semantic interaction fusion module according to an embodiment of the present invention is shown;

[0033] Figure 10 A schematic diagram showing feature aggregation according to an embodiment of the present invention is shown;

[0034] Figure 11 A schematic diagram showing text semantic features according to an embodiment of the present invention is shown;

[0035] Figure 12 A schematic diagram showing second-level image semantic features of a global visual semantic feature extraction model according to an embodiment of the present invention is shown;

[0036] Figure 13 A schematic diagram showing the third-level image semantic features of the global visual semantic feature extraction model according to an embodiment of the present invention is shown;

[0037] Figure 14 A schematic diagram showing fourth-level image semantic features of a global visual semantic feature extraction model according to an embodiment of the present invention is shown;

[0038] Figure 15 A schematic diagram showing first-level high-level semantic features according to an embodiment of the present invention is shown;

[0039] Figure 16 A schematic diagram showing a second-level high-level semantic feature according to an embodiment of the present invention is shown;

[0040] Figure 17 A schematic diagram showing high-level semantic features according to an embodiment of the present invention is shown;

[0041] Figure 18 Schematic diagram showing text enhancement semantic features according to an embodiment of the present invention;

[0042] Figure 19 A schematic diagram showing fusion semantic features according to an embodiment of the present invention is shown;

[0043] Figure 20 A schematic diagram showing local low-level semantic features of a viewport according to an embodiment of the present invention is shown;

[0044] Figure 21 Schematic diagram showing the output K of the fourth convolutional layer according to an embodiment of the present invention;

[0045] Figure 22 Schematic diagram showing an output V of the fifth convolutional layer according to an embodiment of the present invention;

[0046] Figure 23 Schematic diagram showing an output Q of the seventh convolutional layer according to an embodiment of the present invention;

[0047] Figure 24 A schematic diagram showing an output result of a second self-attention module according to an embodiment of the present invention;

[0048] Figure 25 FIG2 shows a block diagram of a panoramic image quality assessment device based on a psychological memory mechanism according to an embodiment of the present invention;

[0049] Figure 26 A block diagram of an exemplary electronic device is shown in which embodiments of the present invention can be implemented.

[0050] Among them, Stem is the spatial transformation embedding module, VSS Block is the visual state space block, Down Sampling is the downsampling module, FFN is the feedforward neural network layer, LN is the normalization layer, SS2D Block is the 2D selective scanning block, 2600 is the electronic device, 2601 is the computing unit, 2602 is the ROM, 2603 is the RAM, 2604 is the bus, 2605 is the I / O interface, 2606 is the input unit, 2607 is the output unit, 2608 is the storage unit, and 2609 is the communication unit. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0053] In this method, textual semantic features and multi-level image semantic features are derived from the distorted panoramic image of the target. High-level semantic features are derived from the textual and multi-level image semantic features. A viewport image is derived from the distorted panoramic image of the target, and low-level semantic features are derived from the viewport image and textual semantic features. The high-level and low-level semantic features are aggregated to obtain a quality assessment score for the distorted panoramic image of the target. This approach, combined with large-scale model technology, fully exploits the perceptual and memory characteristics of the human brain. Furthermore, the incorporation of multimodal features improves existing panoramic image quality assessment algorithms, effectively enhancing the model's overall performance.

[0054] Figure 1 A flowchart of a panoramic image quality assessment method based on a psychological memory mechanism according to an embodiment of the present invention is shown.

[0055] The method includes:

[0056] S101: Obtain a distorted panoramic image of a target.

[0057] Specifically, this example uses the publicly available panoramic image dataset CVIQ to evaluate the proposed model. The dataset contains 528 distorted panoramic images using three different types of compression (JPEG, H.264 / AVC, and H.265 / HEVC). Each distorted image is provided with a subjective score (MOS). All images have a resolution of 4096 × 2048.

[0058] S102: Input the target distorted panoramic image into a text semantic feature extraction model to obtain text semantic features.

[0059] In this embodiment, if Figure 2 As shown, the target distorted panoramic image is input into a text semantic feature extraction model to obtain text semantic features, including:

[0060] S201: Input the target distorted panoramic image into a text generation model, and output a text description sentence of the target distorted panoramic image.

[0061] Specifically, the text generation model is DepictQA (Depicted Image Quality Assessment method, image quality-aware multimodal large language model), which uses MLLM (Multimodal Large Language Model) to provide a human-like, language-based description of image quality.

[0062] As some optional implementations of this embodiment, assume that the target distorted panoramic image is input into the DepictQA model, and the output text description sentence is "The quantization distortion is evident, resulting in aloss of detail and color depth, which affects the images's overall clarity. It has obvious quantization distortion, retains more detail and color accuracy, making it bad to reflect the scene."

[0063] S202: Input the text description of the target distorted panorama into the Imagebind model to obtain text semantic features (such as Figure 11 shown).

[0064] Specifically, in this embodiment, the Imagebind model can embed text data into a unified embedding space, thereby achieving understanding and conversion between different modal data.

[0065] Extracting text description sentences from distorted panoramas can provide non-visual semantic information for the image and improve model accuracy; extracting text semantic information with the help of a large model can reduce computational complexity while obtaining representative text semantic information.

[0066] S103, inputting the target distorted panoramic image into a global visual semantic feature extraction model to obtain multi-level image semantic features; then inputting the text semantic features and the multi-level image semantic features into a global semantic interaction fusion model to obtain high-level semantic features.

[0067] In this embodiment, if Figure 6 As shown, the global visual semantic feature extraction model (VMamba) includes a spatial transformer embedding module (Stem), a first visual state space block (VSS Block), a first downsampling module (Down Sampling), a second visual state space block, a second downsampling module, a third visual state space block, a third downsampling module and a fourth visual state space block.

[0068] The output of the first visual state space block is used as the first-level image semantic feature; the output of the second visual state space block is used as the second-level image semantic feature (such as Figure 12 As shown); the output of the third visual state space block is used as the third-level image semantic feature (as shown Figure 13 As shown); the output of the fourth visual state space block is used as the fourth level image semantic feature (as shown Figure 14 As shown); the first-level image semantic features, the second-level image semantic features, the third-level image semantic features and the fourth-level image semantic features constitute a multi-level image semantic feature.

[0069] In this embodiment, if Figure 7As shown, the first visual state space block includes, in sequence, a third LN (normalization) layer, a first 2D selective scanning block (SS2d Block, 2D-selective-scan Block), a fourth LN layer, and a second FFN (Feed Forward Network, feedforward neural network) layer.

[0070] Specifically, the third LN layer obtains the output of the spatial transformation embedding module, normalizes it, and outputs it to the first 2D selective scanning block for feature extraction, outputting a first extraction result; the output of the spatial transformation embedding module and the first extraction result are added to obtain a first addition result; the first addition result is sequentially input into the fourth LN layer and the second FFN layer to output a first enhancement result; the first addition result and the first enhancement result are added to obtain the output of the first visual state space block.

[0071] In this embodiment, the second visual state space block sequentially includes a fifth LN layer, a second 2D selective scanning block, a sixth LN layer, and a third FFN layer.

[0072] Specifically, the fifth LN layer obtains the output of the first downsampling module, normalizes it, and outputs it to the second 2D selective scanning block for feature extraction, and outputs a second extraction result; the output of the first downsampling module and the second extraction result are added to obtain a second addition result; the second addition result is sequentially input into the sixth LN layer and the third FFN layer, and a second enhancement result is output; the second addition result and the second enhancement result are added to obtain the output of the second visual state space block.

[0073] In this embodiment, the third visual state space block sequentially includes the seventh LN layer, the third 2D selective scanning block, the eighth LN layer and the fourth FFN layer.

[0074] Specifically, the seventh LN layer obtains the output of the second downsampling module, normalizes it, and outputs it to the third 2D selective scanning block for feature extraction, and outputs a third extraction result; the output of the second downsampling module and the third extraction result are added to obtain a third addition result; the third addition result is sequentially input into the eighth LN layer and the fourth FFN layer, and a third enhancement result is output; the third addition result and the third enhancement result are added to obtain the output of the third visual state space block.

[0075] In this embodiment, the fourth visual state space block sequentially includes a ninth LN layer, a fourth 2D selective scanning block, a tenth LN layer, and a fifth FFN layer.

[0076] Specifically, the ninth LN layer obtains the output of the third downsampling module, normalizes it, and outputs it to the fourth 2D selective scanning block for feature extraction, and outputs a fourth extraction result; the output of the third downsampling module and the fourth extraction result are added to obtain a fourth addition result; the fourth addition result is sequentially input into the tenth LN layer and the fifth FFN layer, and a fourth enhancement result is output; the fourth addition result and the fourth enhancement result are added to obtain the output of the fourth visual state space block.

[0077] The VMamba model can effectively obtain the long-range dependencies and local details of the image. Its hierarchical structure can effectively capture rich multi-scale semantic information with low computational complexity.

[0078] In this embodiment, if Figure 3 、 8 As shown, the text semantic features and the multi-level image semantic features are input into the global semantic interaction fusion model to obtain high-level semantic features, including:

[0079] S301, the text semantic features are sequentially input into the first convolutional layer Conv and the first ReLu activation function; then the output of the first ReLu activation function and the second-level image semantic features S2 are used as the input of the first LSTM module, and the first-level high-level semantic features (such as Figure 15 shown).

[0080] S302, the text semantic features are sequentially input into the second convolutional layer Conv and the second ReLu activation function; the output of the second ReLu activation function, the third-level image semantic features S3 and the first-level high-level semantic features are then used as the input of the second LSTM module, and the second-level high-level semantic features (such as Figure 16 shown).

[0081] S303, the text semantic features are sequentially input into the third convolutional layer Conv and the third ReLu activation function; the output of the third ReLu activation function, the fourth-level image semantic features S4 and the second-level high-level semantic features are then used as inputs of the third LSTM module, the third-level high-level semantic features are output, and the third-level high-level semantic features are used as high-level semantic features (such as Figure 17 shown).

[0082] Specifically, the convolution kernel size of the first convolution layer, the second convolution layer, and the third convolution layer are all 6, and the stride is 6.

[0083] Taking into account the descriptive and non-descriptive memory features in the human brain, LSTM is used to perform multi-level interactive fusion of text semantics and multi-level image semantic information, effectively simulating the fusion strategy of the human brain's memory mechanism, and can obtain accurate and rich high-level semantic information.

[0084] S104, performing viewport processing on the target distorted panoramic image to obtain a plurality of viewport images; inputting each viewport image and the text semantic features into a local semantic interaction fusion model respectively, outputting the local low-level semantic features of each viewport, and then splicing the local low-level semantic features of all viewports to obtain low-level semantic features.

[0085] Specifically, the viewport acquisition method uses FoV Selection (Field of View Selection) to acquire a viewport image of the target distorted panorama.

[0086] In this embodiment, the local semantic interaction fusion model includes several semantic interaction fusion modules, and each semantic interaction fusion module corresponds to a viewport, which is used to take the viewport map of the corresponding viewport and the text semantic features as input, and output the local low-level semantic features of each viewport; each semantic interaction fusion module includes a first self-attention module, a first cross-attention module, a first FFN layer and a viewport semantic feature extraction module.

[0087] In this embodiment, if Figure 4 、 9 As shown, the viewport map of the corresponding viewport and the text semantic features are used as input to output local low-level semantic features of each viewport, including:

[0088] S401, input the text semantic feature into the first self-attention module (SA), add the output of the first self-attention module to the text semantic feature, and obtain a text enhanced semantic feature (such as Figure 18 shown).

[0089] S402: Input the viewport graph of the corresponding viewport into the viewport semantic feature extraction module (ViT, VisionTransformer) to obtain viewport semantic features; then use the viewport semantic features and the text enhancement semantic features as inputs of the first cross-attention module (CA, Cross-Attention), and output fused semantic features (such as Figure 19 shown).

[0090] S403, adding the text enhanced semantic features and the fused semantic features and inputting them into the first FFN layer to obtain enhanced fused semantic features; then adding the enhanced fused semantic features and the input of the first FFN layer to obtain local low-level semantic features of each viewport (such as Figure 20 shown).

[0091] Viewport segmentation of distorted panoramas is consistent with human visual perception characteristics and helps extract local visual semantic features; text information is fused with each viewport separately to mine rich low-level semantic information, which complements high-level semantic information and improves the overall performance of the model; feature fusion using self-attention and cross-attention modules can capture semantic interactions within and between modalities, improving the overall performance of the model.

[0092] S105, such as Figure 5 、 10 As shown, the high-level semantic features and the low-level semantic features are aggregated to obtain a quality evaluation score of the target distorted panoramic image, including:

[0093] S501, inputting the high-level semantic features into the first LN layer; inputting the output of the first LN layer into the fourth convolutional layer Conv and the fifth convolutional layer Conv respectively; and

[0094] The low-level semantic features are input into the sixth convolutional layer Conv; the output of the sixth convolutional layer Conv is input into the second LN layer; and the output of the second LN layer is used as the input of the seventh convolutional layer Conv.

[0095] S502, the output K of the fourth convolutional layer (such as Figure 21 As shown), the output V of the fifth convolutional layer (as shown Figure 22 ) and the output Q of the seventh convolutional layer (as Figure 23 ) is input into the second self-attention module, and then the output of the second self-attention module (as shown Figure 24 As shown in Figure 3, the first fully connected layer (FC) is used as the input to output the quality evaluation score (Score) of the target distorted panoramic image.

[0096] As some optional implementations of this embodiment, feature aggregation includes:

[0097]

[0098] in, is the output of the second self-attention module; It is a low-level semantic feature; It is a high-level semantic feature; is the query weight matrix; is the transpose of the key weight matrix; is the transposition of high-level semantic features; To prevent gradient messages from introducing parameters; is the value weight matrix.

[0099] As some optional implementation methods of this embodiment, the evaluation indicators used are PLCC (Pearson linear correlation coefficient), SRCC (Spearman's Rank Correlation Coefficient) and RMSE (Root Mean Square Error), where the higher the PLCC and SRCC values and the smaller the RMSE value, the better the model performance.

[0100] The PLCC calculation formula is as follows:

[0101]

[0102]

[0103]

[0104] in, is the number of test images; is the index value of the number of test images; is the mean subjective quality score of all images in the prediction dataset; Predict the mean quality score for the model; The first test image The subjective quality scores provided by the datasets; is the quality score predicted by the model; Linear consistency between the predicted image quality scores and the quality scores provided by the dataset.

[0105] The calculation formula is as follows:

[0106]

[0107] in, Monotonicity between the predicted quality scores and the subjective scores provided by the dataset.

[0108] The RMSE calculation formula is as follows:

[0109]

[0110] Here, RMSE is the error between the predicted score and the subjective score.

[0111] In this embodiment, the subjective score MOS value provided by the data set is And the objective quality score obtained by the present invention Input PLCC, SRCC and RMSE calculation formulas to obtain evaluation index values, and then replace the above two modules with simple splicing and fusion modules respectively to calculate the evaluation index values. The results are shown in Table 1 below. It can be seen from the above results that the PLCC and SRCC of the present invention are closest to 1, and the RMSE is the smallest, indicating that the fusion method of the present invention can effectively enhance semantic features and improve feature representation capabilities, and can also achieve effective fusion of features and improve model accuracy. It also proves that the present invention can accurately evaluate the quality of panoramic images.

[0112] Table 1

[0113] PLCC SRCC RMSE Model of the present invention 0.9625 0.9639 2.7753 Replace the global semantic interaction fusion with the local semantic interaction fusion module 0.9517 0.9501 3.5124 Feature aggregation model replaced by splicing model 0.9501 0.9579 3.0325

[0114] The interactive fusion of high-level semantics and low-level semantics can further refine semantic features and improve the feature representation ability of the model; using the self-attention model for feature aggregation can capture the semantic interaction relationship between modalities and improve the overall performance of the model.

[0115] According to the embodiments of the present invention, the present invention has the following advantages and effects compared with the prior art:

[0116] (1) Based on the human brain memory mechanism, the multi-level interactive fusion of text information and visual information is realized, which can deeply explore the interactive relationship between the two modalities, extract rich high-level semantic features, and improve the feature representation ability of the model.

[0117] (2) Considering that different viewports contain different local semantic information for the overall perceptual quality and contribute differently to the overall quality, interactive fusion of them with text information can mine the low-level semantic information contained in the local viewport, effectively complement the high-level semantic information, and improve the overall performance of the model.

[0118] (3) The feature aggregation of high-level semantic features and low-level semantic features can further refine the semantic features extracted by the model and further improve the accuracy of the model compared with simple splicing.

[0119] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0120] The above is an introduction to the method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment having the same inventive concept as the method in the aforementioned embodiment.

[0121] like Figure 25 As shown, the apparatus 2500 includes:

[0122] The acquisition module 2510 is configured to acquire a distorted panoramic image of the target.

[0123] The first feature extraction module 2520 is configured to input the target distorted panoramic image into a text semantic feature extraction model to obtain text semantic features.

[0124] The second feature extraction module 2530 is used to input the target distorted panoramic image into a global visual semantic feature extraction model to obtain multi-level image semantic features; and then input the text semantic features and the multi-level image semantic features into a global semantic interaction fusion model to obtain high-level semantic features.

[0125] The third feature extraction module 2540 is used to perform viewport processing on the target distorted panoramic image to obtain a plurality of viewport images; each viewport image and the text semantic features are input into the local semantic interaction fusion model respectively, and the local low-level semantic features of each viewport are output; then the local low-level semantic features of all viewports are spliced to obtain low-level semantic features.

[0126] The evaluation module 2550 is configured to perform feature aggregation on the high-level semantic features and the low-level semantic features to obtain a quality evaluation score of the target distorted panoramic image.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0128] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0129] According to an embodiment of the present invention, the present invention further provides an electronic device and a readable storage medium.

[0130] Figure 26 A schematic block diagram of an electronic device 2600 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0131] Electronic device 2600 includes a computing unit 2601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2602 or a computer program loaded from a storage unit 2608 into a random access memory (RAM) 2603. RAM 2603 may also store various programs and data required for the operation of electronic device 2600. Computing unit 2601, ROM 2602, and RAM 2603 are interconnected via a bus 2604. An input / output (I / O) interface 2605 is also connected to bus 2604.

[0132] Multiple components in electronic device 2600 are connected to I / O interface 2605, including: an input unit 2606, such as a keyboard, mouse, etc.; an output unit 2607, such as various types of displays, speakers, etc.; a storage unit 2608, such as a magnetic disk, optical disk, etc.; and a communication unit 2609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 2609 allows electronic device 2600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0133] Computing unit 2601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 2601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 2601 performs the various methods and processes described above, such as methods S101 through S105. For example, in some embodiments, methods S101 through S105 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 2608. In some embodiments, part or all of the computer program can be loaded and / or installed onto electronic device 2600 via ROM 2602 and / or communication unit 2609. When the computer program is loaded into RAM 2603 and executed by computing unit 2601, one or more steps of methods S101 through S105 described above can be performed. Alternatively, in other embodiments, the computing unit 2601 may be configured to execute methods S101 to S105 in any other appropriate manner (eg, by means of firmware).

[0134] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0139] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0140] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0141] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A panoramic image quality evaluation method based on psychological memory mechanism, characterized in that: include: Obtain a distorted panoramic image of the target; Inputting the target distorted panoramic image into a text semantic feature extraction model to obtain text semantic features; Inputting the target distorted panoramic image into a global visual semantic feature extraction model to obtain multi-level image semantic features; Then, the text semantic features and the multi-level image semantic features are input into the global semantic interaction fusion model to obtain high-level semantic features, including: The text semantic features are sequentially input into the first convolutional layer and the first ReLu activation function; the output of the first ReLu activation function and the second-level image semantic features are then used as inputs of the first LSTM module to output first-level high-level semantic features; the text semantic features are sequentially input into the second convolutional layer and the second ReLu activation function; the output of the second ReLu activation function, the third-level image semantic features and the first-level high-level semantic features are then used as inputs of the second LSTM module to output second-level high-level semantic features; the text semantic features are sequentially input into the third convolutional layer and the third ReLu activation function; the output of the third ReLu activation function, the fourth-level image semantic features and the second-level high-level semantic features are then used as inputs of the third LSTM module to output third-level high-level semantic features, and the third-level high-level semantic features are used as high-level semantic features; The target distorted panoramic image is subjected to viewport processing to obtain a plurality of viewport images; each viewport image and the text semantic features are input into a local semantic interaction fusion model respectively, and a local low-level semantic feature of each viewport is output; and the local low-level semantic features of all viewports are then spliced to obtain a low-level semantic feature; The local semantic interaction fusion model includes several semantic interaction fusion modules, and each semantic interaction fusion module corresponds to a viewport, and is used to take the viewport map of the corresponding viewport and the text semantic features as input and output the local low-level semantic features of each viewport; each semantic interaction fusion module includes a first self-attention module, a first cross-attention module, a first FFN layer and a viewport semantic feature extraction module; The method takes the viewport graph of the corresponding viewport and the text semantic features as input and outputs the local low-level semantic features of each viewport, including: Input the text semantic feature into the first self-attention module, add the output of the first self-attention module to the text semantic feature to obtain a text enhanced semantic feature; input the viewport map of the corresponding viewport into the viewport semantic feature extraction module to obtain a viewport semantic feature; then use the viewport semantic feature and the text enhanced semantic feature as input to the first cross attention module, outputting a fused semantic feature; add the text enhanced semantic feature and the fused semantic feature and input them into the first FFN layer to obtain an enhanced fused semantic feature; then add the enhanced fused semantic feature to the input of the first FFN layer to obtain a local low-level semantic feature of each viewport; The high-level semantic features and the low-level semantic features are aggregated to obtain a quality evaluation score of the target distorted panoramic image.

2. The method according to claim 1, characterized in that The step of inputting the target distorted panoramic image into a text semantic feature extraction model to obtain text semantic features includes: Inputting the target distorted panoramic image into a text generation model, and outputting a text description sentence of the target distorted panoramic image; The text description of the target distorted panorama is input into the Imagebind model to obtain text semantic features.

3. The method according to claim 1, characterized in that The global visual semantic feature extraction model includes, in sequence, a spatial transformation embedding module, a first visual state space block, a first downsampling module, a second visual state space block, a second downsampling module, a third visual state space block, a third downsampling module, and a fourth visual state space block; Among them, the output of the first visual state space block is used as the first-level image semantic feature; the output of the second visual state space block is used as the second-level image semantic feature; the output of the third visual state space block is used as the third-level image semantic feature; the output of the fourth visual state space block is used as the fourth-level image semantic feature; the first-level image semantic feature, the second-level image semantic feature, the third-level image semantic feature and the fourth-level image semantic feature constitute a multi-level image semantic feature.

4. The method according to claim 1, wherein The step of aggregating the high-level semantic features and the low-level semantic features to obtain a quality evaluation score of the target distorted panoramic image includes: Inputting the high-level semantic features into the first LN layer; inputting the output of the first LN layer into the fourth convolutional layer and the fifth convolutional layer respectively; and Inputting the low-level semantic features into the sixth convolutional layer; inputting the output of the sixth convolutional layer into the second LN layer; and then using the output of the second LN layer as the input of the seventh convolutional layer; The outputs of the fourth, fifth, and seventh convolutional layers are input into the second self-attention module, and then the output of the second self-attention module is used as the input of the first fully connected layer to output the quality evaluation score of the target distorted panoramic image.

5. A panoramic image quality assessment device based on psychological memory mechanism, characterized in that: include: An acquisition module, used to obtain a distorted panoramic image of the target; A first feature extraction module is configured to input the target distorted panoramic image into a text semantic feature extraction model to obtain text semantic features; A second feature extraction module is used to input the target distorted panoramic image into a global visual semantic feature extraction model to obtain multi-level image semantic features; Then, the text semantic features and the multi-level image semantic features are input into the global semantic interaction fusion model to obtain high-level semantic features, including: The text semantic features are sequentially input into the first convolutional layer and the first ReLu activation function; the output of the first ReLu activation function and the second-level image semantic features are then used as inputs of the first LSTM module to output first-level high-level semantic features; the text semantic features are sequentially input into the second convolutional layer and the second ReLu activation function; the output of the second ReLu activation function, the third-level image semantic features and the first-level high-level semantic features are then used as inputs of the second LSTM module to output second-level high-level semantic features; the text semantic features are sequentially input into the third convolutional layer and the third ReLu activation function; the output of the third ReLu activation function, the fourth-level image semantic features and the second-level high-level semantic features are then used as inputs of the third LSTM module to output third-level high-level semantic features, and the third-level high-level semantic features are used as high-level semantic features; A third feature extraction module is configured to perform viewport processing on the target distorted panoramic image to obtain a plurality of viewport images; input each viewport image and the text semantic features into a local semantic interaction fusion model, output a local low-level semantic feature of each viewport, and then splice the local low-level semantic features of all viewports to obtain a low-level semantic feature; The local semantic interaction fusion model includes several semantic interaction fusion modules, and each semantic interaction fusion module corresponds to a viewport, and is used to take the viewport map of the corresponding viewport and the text semantic features as input and output the local low-level semantic features of each viewport; each semantic interaction fusion module includes a first self-attention module, a first cross-attention module, a first FFN layer and a viewport semantic feature extraction module; The method takes the viewport graph of the corresponding viewport and the text semantic features as input and outputs the local low-level semantic features of each viewport, including: Input the text semantic feature into the first self-attention module, add the output of the first self-attention module to the text semantic feature to obtain a text enhanced semantic feature; input the viewport map of the corresponding viewport into the viewport semantic feature extraction module to obtain a viewport semantic feature; then use the viewport semantic feature and the text enhanced semantic feature as input to the first cross attention module, outputting a fused semantic feature; add the text enhanced semantic feature and the fused semantic feature and input them into the first FFN layer to obtain an enhanced fused semantic feature; then add the enhanced fused semantic feature to the input of the first FFN layer to obtain a local low-level semantic feature of each viewport; An evaluation module is used to perform feature aggregation on the high-level semantic features and the low-level semantic features to obtain a quality evaluation score of the target distorted panoramic image.

6. An electronic device comprising at least one processor; and a memory communicatively connected to the at least one processor; characterized in that: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.

Images