Video data processing method and device based on left and right views, equipment and medium

Abstract

The invention provides a video data processing method and device based on left and right views, equipment and a medium. The method comprises the following steps: determining left view data and right view data of a to-be-processed video; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view; determining parallax fusion features of the to-be-processed video according to the left view data and the right view data; wherein the parallax fusion feature represents views of the left eye and the right eye and disparity maps of the left eye and the right eye; determining a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description; wherein the preset text description represents the evaluation dimension of the to-be-processed video, and the quality score represents the video quality of the to-be-processed video. According to the method, the parallax fusion feature of the to-be-processed video is determined through the left view data and the right view data, and the parallax fusion feature of the to-be-processed video can improve the accuracy of data processing.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device, equipment and medium for processing video data based on left and right views. Background Art

[0002] Stereoscopic video (i.e. 3D video) is widely used in fields such as virtual reality, mixed reality and holographic projection. Its three-dimensional depth perception characteristics far exceed those of two-dimensional video and are closer to the real human viewing experience.

[0003] However, stereoscopic video is susceptible to noise interference during the construction, encoding, transmission and display processes, which affects the video quality and causes visual discomfort. It is crucial to be able to reasonably evaluate and thus improve stereoscopic video. Therefore, accurately evaluating stereoscopic video is an issue that urgently needs to be solved. Summary of the invention

[0004] The present application provides a method, device, equipment and medium for processing video data based on left and right views, so as to solve the problem of accurately evaluating stereoscopic video.

[0005] In a first aspect, the present application provides a method for processing video data based on left and right views, comprising:

[0006] Determine left view data and right view data of the video to be processed; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view;

[0007] Determine the disparity fusion feature of the video to be processed according to the left view data and the right view data; wherein the disparity fusion feature represents the views of the left and right eyes and the disparity maps of the left and right eyes;

[0008] The quality score of the video to be processed is determined according to the disparity fusion feature and the preset text description; wherein the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0009] Optionally, the above method, according to the left view data and the right view data, determines the disparity fusion feature of the video to be processed, including:

[0010] Obtain disparity map data according to the left view data and the right view data;

[0011] The disparity fusion features of the video to be processed are determined according to the left view data, the right view data, and the disparity map data.

[0012] Optionally, the above method, according to the left view data, the right view data, and the disparity map data, determines the disparity fusion features of the video to be processed, including:

[0013] Encoding the left view data, the right view data, and the disparity data respectively to obtain a feature vector of the left view data, a feature vector of the right view data, and a feature vector of the disparity data;

[0014] The disparity fusion feature of the video to be processed is obtained according to the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data.

[0015] Optionally, the above method obtains the disparity fusion feature of the video to be processed according to the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data, including:

[0016] Aligning the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data to obtain an alignment vector of the left view data, an alignment vector of the right view data, and an alignment vector of the disparity map data; wherein the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data are located in the same coordinate system;

[0017] The disparity fusion feature of the video to be processed is obtained according to the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data.

[0018] Optionally, the above method obtains the disparity fusion feature of the video to be processed according to the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data, including:

[0019] Determining a similarity between an alignment vector of the left view data and an alignment vector of the right view data;

[0020] According to the similarity, determine the weight of the left view data as a first weight, and determine the weight of the right view data as a second weight;

[0021] Determine the weight of the disparity map according to the alignment vector of the disparity map data, which is the third weight;

[0022] A disparity fusion feature of the video to be processed is determined according to an alignment vector of the left view data, an alignment vector of the right view data, an alignment vector of the disparity map data, a first weight, a second weight, and a third weight.

[0023] Optionally, the above method, according to the similarity, determines the weight of the left view data as a first weight, and determines the weight of the right view data as a second weight, including:

[0024] Obtaining a first score according to the similarity and a preset first parameter; wherein the first score represents a consistency score of the left view data relative to the right view data;

[0025] Obtaining a second score according to the similarity and a preset second parameter; wherein the second score represents a consistency score of the right view data relative to the left view data;

[0026] According to the first score, the weight of the left view data is obtained, which is the first weight, and according to the second score, the weight of the right view data is determined, which is the second weight.

[0027] Optionally, the above method, according to the parallax fusion feature and the preset text description, determines the quality score of the video to be processed, including:

[0028] The disparity fusion features and the preset text description are input into the preset large model to obtain the quality score of the output video to be processed.

[0029] Optionally, the above method further comprises:

[0030] Obtain a data set to be trained; wherein the data set to be trained includes a video to be trained, label data of the video to be trained, the label data representing an actual quality score of the video to be trained, and a text description to be trained;

[0031] The initial model is trained according to the data set to be trained to obtain a trained video scoring model; wherein the video scoring model is used to determine the left view data and the right view data of the video to be processed, and the disparity fusion features of the video to be processed are determined according to the left view data and the right view data; the quality score of the video to be processed is determined according to the disparity fusion features and a preset text description.

[0032] Optionally, the above method trains the initial model according to the training data set to obtain a trained video scoring model, including:

[0033] Input the video to be trained into the initial model to obtain the fused feature vector of the video to be trained;

[0034] Converting the fused feature vector of the video to be trained into one-dimensional data, and determining the score difference between the one-dimensional data and the actual quality score of the video to be trained;

[0035] Determine the left view data and the right view data of the video to be trained, and obtain a first disparity loss value and a second disparity loss value according to the left view data and the right view data of the video to be trained; wherein the first disparity loss value represents the disparity of the left view data of the video to be trained relative to the right view data, and the second disparity loss value represents the disparity of the right view data of the video to be trained relative to the left view data;

[0036] Determine a difference between the first parallax loss value and the second parallax loss value as a loss difference;

[0037] Determine the feature difference between the fused feature vectors of adjacent frames in the video to be trained; wherein the feature difference represents the change between the fused feature vectors of adjacent frames;

[0038] If the score difference, loss difference, and feature difference all meet the preset difference conditions, it is determined that the initial model training is completed and the video scoring model is obtained.

[0039] In a second aspect, the present application provides a device for processing video data based on left and right views, comprising:

[0040] A first determination module is used to determine left view data and right view data of the video to be processed; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view;

[0041] A second determination module is used to determine the disparity fusion feature of the video to be processed according to the left view data and the right view data; wherein the disparity fusion feature represents the views of the left and right eyes and the disparity maps of the left and right eyes;

[0042] The third determination module is used to determine the quality score of the video to be processed according to the disparity fusion feature and the preset text description; wherein the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0043] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0044] Memory stores computer-executable instructions;

[0045] The processor executes the computer-executable instructions stored in the memory to implement the method in the present application.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method of the present application.

[0047] In a fifth aspect, the present application provides a computer program product, comprising: a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementations of the first aspect.

[0048] The present application provides a method, device, equipment and medium for processing video data based on left and right views, which determines the left view data and right view data of the video to be processed; wherein the left view data is the image frame of the left eye view, and the right view data is the image frame of the right eye view; based on the left view data and the right view data, the disparity fusion feature of the video to be processed is determined; wherein the disparity fusion feature represents the views of the left and right eyes and the disparity map of the left and right eyes; through the left eye view number and the right view data, the fusion feature is obtained, and a comprehensive evaluation can be obtained for the quality issues raised by the stereoscopic video. According to the disparity fusion feature and the preset text description, the quality score of the video to be processed is determined; through the disparity fusion feature and the preset text description, a variety of data can be considered to obtain a more accurate, detailed and comprehensive evaluation, thereby obtaining the quality score of the video to be processed. wherein, the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 A flowchart of a method for processing video data based on left and right views provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of another method for processing video data based on left and right views provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of another method for processing video data based on left and right views provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the structure of a device for processing video data based on left and right views provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the structural architecture of a device for processing video data based on left and right views provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0056] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0058] At present, the existing evaluation of stereoscopic videos mainly relies on extracting objective features of images and trying to fit or approximate human subjective scores through these features. However, these methods are limited by their single-dimensional analysis framework and focus on improving one or several image features. The quality of stereoscopic videos is affected by many complex factors. These methods often find it difficult to fully and accurately reflect the true quality of stereoscopic videos.

[0059] The method for processing video data based on left and right views provided in the present application is intended to solve the above technical problems in the prior art.

[0060] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0061] The present application provides a method, device, equipment and medium for processing video data based on left and right views, which determines the left view data and right view data of the video to be processed; wherein the left view data is the image frame of the left eye view, and the right view data is the image frame of the right eye view; based on the left view data and the right view data, the disparity fusion feature of the video to be processed is determined; wherein the disparity fusion feature represents the views of the left and right eyes and the disparity map of the left and right eyes; through the left eye view number and the right view data, the fusion feature is obtained, and a comprehensive evaluation can be obtained for the quality issues raised by the stereoscopic video. According to the disparity fusion feature and the preset text description, the quality score of the video to be processed is determined; through the disparity fusion feature and the preset text description, a variety of data can be considered to obtain a more accurate, detailed and comprehensive evaluation, thereby obtaining the quality score of the video to be processed. wherein, the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0062] Figure 1 The flowchart of a method for processing video data based on left and right views provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the execution subject of the method can be a server or other servers, and this embodiment is not particularly limited here. Figure 1As shown, the method includes:

[0063] S101. Determine left view data and right view data of a video to be processed; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view.

[0064] The left view refers to the view observed by the left eye.

[0065] The right view refers to the view observed by the right eye.

[0066] The video to be processed is a 3D video. Specific frames are extracted from the video stream of the video to be processed by FBS (Frame Blending Sampling Technique) to obtain left view data and right view data. The extracted frames can be at equal intervals or based on specific events or changes (such as scene switching).

[0067] S102 . Determine a disparity fusion feature of the video to be processed according to the left view data and the right view data; wherein the disparity fusion feature represents the views of the left and right eyes and the disparity maps of the left and right eyes.

[0068] Among them, a disparity map is obtained according to the left view data and the right view data, and then the left view, the right view and the disparity map are fused to obtain the disparity fusion feature of the video to be processed.

[0069] For example, the disparity map is obtained by combining the left view data, the right view data, and a preset disparity calculation algorithm (RAFT-Stereo), and the preset disparity algorithm may be a stereo matching algorithm (RAFT-Stereo).

[0070] The fusion processing can assign different weights to different left image feature coding sequences, right image feature coding sequences, and disparity map feature coding sequences, and then obtain the disparity fusion features of the video to be processed by weighted summation.

[0071] In the embodiment of the present application, the parallax fusion feature of the video to be processed is determined according to the left view data and the right view data, including:

[0072] Obtain disparity map data according to the left view data and the right view data;

[0073] The disparity fusion features of the video to be processed are determined according to the left view data, the right view data, and the disparity map data.

[0074] Among them, the parallax map is an image technology used to calculate and represent depth information, and is commonly used in computer vision, 3D graphics rendering, virtual reality and other fields. The parallax map helps generate images with a sense of three-dimensionality and depth by recording the depth information of different points in the scene. The parallax map represents the displacement of each pixel between the left view and the right view.

[0075] To obtain disparity data from the left view data and the right view data, it is necessary to first ensure that the left view data and the right view data are on the same horizontal line, and then calculate the matching cost for each pixel of the left view data and the right view data. The matching cost is used to compare the similarity of the pixels in the left view and the right view, and then find the best match within the disparity range based on the similarity of the pixels in the left view and the right view. For example, for block matching: select a small block (usually a rectangular area) in one image (for example, the left view), and then search for matching blocks in another image (for example, the right view). The matching standard is usually based on some similarity metric, such as the sum of absolute differences, the sum of squared differences, or normalized cross-correlation. The "StereoBM" class provided by OpenCV (Open Source Computer Vision Library) for block matching can be used to obtain disparity map data, and then the disparity map data is post-processed (i.e., the disparity map is smoothed and noise removed) to obtain a higher quality disparity map. Disparity map data can also be obtained through a stereo matching algorithm, left view data, and right view data.

[0076] The disparity map data of the left view data and the right view data are calculated through the stereo matching algorithm (RAFT-Stereo), and then different weights are assigned to the left view data, the right view data, and the disparity map data. Then, the weighted sum is performed to obtain the disparity fusion features of the video to be processed. Different weights can be calculated through preset formulas or preset by the user.

[0077] The benefit of this setting is that by generating a disparity map, the depth information in the scene can be captured more accurately. This makes the depth perception more realistic and natural when watching the video on a 3D display device. The disparity fusion feature takes into account the left view data, right view data, and disparity map data, making the evaluation more accurate.

[0078] In the embodiment of the present application, the disparity fusion features of the video to be processed are determined according to the left view data, the right view data, and the disparity map data, including:

[0079] Encoding the left view data, the right view data, and the disparity data respectively to obtain a feature vector of the left view data, a feature vector of the right view data, and a feature vector of the disparity data;

[0080] The disparity fusion feature of the video to be processed is obtained according to the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data.

[0081] Among them, encoding processing refers to feature encoding, which is a commonly used technique in data preprocessing, especially in the fields of machine learning and data mining. Its purpose is to convert raw data into a numerical format suitable for processing by machine learning algorithms. Feature encoding is particularly important for processing non-numerical data (such as categorical data).

[0082] In machine learning, a feature vector usually refers to a vector used to describe a data instance. Each element represents a feature or attribute. For example, in image processing, a feature vector may contain pixel values, color histograms, or other image features.

[0083] A visual encoder is used to perform feature encoding on the left view data, the right view data, and the disparity data, respectively, to obtain a left image feature encoding sequence (i.e., a feature vector of the left view data), a right image feature encoding sequence (i.e., a feature vector of the right view data), and a disparity map feature encoding sequence (i.e., a feature vector of the disparity map data). The left image feature encoding sequence, the right image feature encoding sequence, and the disparity map feature encoding sequence are fused. The fusion processing can assign different weights to different left image feature encoding sequences, right image feature encoding sequences, and disparity map feature encoding sequences, and then the disparity fusion features of the video to be processed are obtained by weighted summation.

[0084] The advantage of this setting is that the left view data, right view data, and disparity map data can be better integrated through encoding processing. By integrating the features of the left view, right view, and disparity map, the depth information of the scene can be captured more accurately, thereby improving the accuracy of depth perception. Through feature fusion, the noise and errors that may exist in a single view or disparity map can be effectively reduced, improving the reliability of the overall data.

[0085] In the embodiment of the present application, the disparity fusion feature of the video to be processed is obtained according to the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data, including:

[0086] Aligning the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data to obtain an alignment vector of the left view data, an alignment vector of the right view data, and an alignment vector of the disparity map data; wherein the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data are located in the same coordinate system;

[0087] The disparity fusion feature of the video to be processed is obtained according to the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data.

[0088] Among them, the alignment processing usually refers to selecting the feature vector of the disparity map data as the reference coordinate system, that is, converting the feature vector of the left view data and the feature vector of the right view data to the disparity feature coordinate system. Specifically, assuming that the feature vector range of the disparity map data is between [-1, 1] and the feature vector range of the left view data is between [0, 255], and unifying them to the same coordinate system means converting the dimension of the feature vector value of the left view data to between [-1, 1].

[0089] The feature vector of the disparity map data can be selected as the reference coordinate system, and the feature vector of the left view data and the feature vector of the right view data can be converted into the disparity feature coordinate system to obtain the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data. The alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data are fused to obtain the disparity fusion features of the video to be processed.

[0090] The benefit of this setting is that by aligning data from different views to the same coordinate system, the feature inconsistency caused by perspective differences is reduced, thereby improving the accuracy of feature fusion. The alignment process reduces the uncertainty caused by perspective changes and parallax errors, making subsequent video processing more robust and stable.

[0091] In the embodiment of the present application, the disparity fusion feature of the video to be processed is obtained according to the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data, including:

[0092] Determining a similarity between an alignment vector of the left view data and an alignment vector of the right view data;

[0093] According to the similarity, determine the weight of the left view data as a first weight, and determine the weight of the right view data as a second weight;

[0094] Determine the weight of the disparity map according to the alignment vector of the disparity map data, which is the third weight;

[0095] A disparity fusion feature of the video to be processed is determined according to an alignment vector of the left view data, an alignment vector of the right view data, an alignment vector of the disparity map data, a first weight, a second weight, and a third weight.

[0096] The similarity can refer to cosine similarity or cosine distance. Cosine distance is complementary to cosine similarity and is calculated as 1 minus cosine similarity.

[0097] The cosine similarity is:

[0098]

[0099] Where (x, y) represents any position, F L Indicates the alignment vector of the left view data, F R represents the alignment vector of the right view data, and S(x, y) is the cosine similarity.

[0100] The cosine distance is:

[0101] s(x, y) = 1 - S(x, y)

[0102] Among them, s(x, y) is the cosine distance.

[0103] α LR (x, y)=exp(-λ·|d(x, y)|)

[0104] α RL (x, y)=exp(-μ·|d(x, y)|)

[0105] Among them, d(x, y) is the alignment vector of the disparity map data at position (x, y), α LR , α RL are different directional weights, which can be preset constants or parameters learned through the network. λ and μ are adjustment parameters that control the weights to change with the disparity value.

[0106] s′ LR (x, y) = s(x, y)·α LR (x, y)

[0107] Among them, s′ LR (x, y) is the consistency score of the alignment vector of the left view data.

[0108] s′ RL (x, y) = s(x, y)·α RL (x, y)

[0109] Among them, s′ RL (x, y) is the consistency score of the alignment vector of the right view data.

[0110] The first weight acquisition method is:

[0111]

[0112] Among them, W L (x, y) is the first weight, and i, j represent the view traversal position.

[0113] The second weight acquisition method is:

[0114]

[0115] Among them, W R (x, y) is the second weight.

[0116] The third weight acquisition method is:

[0117]

[0118] Among them, W D (x, y) is the third weight.

[0119] The parallax fusion features of the video to be processed are:

[0120] F f (x, y) = W L (x, y)×F L (x,y)+W R (x, y)×F R (x,y)+W D (x, y)×F D (x, y)

[0121] Among them, F L (x, y) is the alignment vector of the left view data, F R (x, y) is the alignment vector of the right view,

[0122] F D (x, y)

[0123] is the alignment vector of the disparity map data.

[0124] Confirm the cosine similarity of the alignment vector of the left view data and the alignment vector of the right view data, determine the cosine distance based on the cosine similarity, substitute the cosine distance into a preset formula to obtain the weight of the left view data and the weight of the right view data, obtain the alignment vector of the disparity map data based on the alignment vector of the disparity map data, and obtain the disparity fusion features of the video to be processed by combining the alignment vector of the left view data, the alignment vector of the right view data, the alignment vector of the disparity map data, the weight of the left view data, the weight of the right view data, and the weight of the disparity map.

[0125] The advantage of this setting is that through precise similarity calculation and weight distribution, efficient disparity fusion feature extraction is achieved, which significantly improves the effect and quality of video processing.

[0126] In the embodiment of the present application, according to the similarity, determining the weight of the left view data as the first weight, and determining the weight of the right view data as the second weight, includes:

[0127] Obtaining a first score according to the similarity and a preset first parameter; wherein the first score represents a consistency score of the left view data relative to the right view data;

[0128] Obtaining a second score according to the similarity and a preset second parameter; wherein the second score represents a consistency score of the right view data relative to the left view data;

[0129] According to the first score, the weight of the left view data is obtained, which is the first weight, and according to the second score, the weight of the right view data is determined, which is the second weight.

[0130] Among them, the first parameter is preset as α LR (x, y), the second parameter is preset to α RL , different directional weights can be preset constants. The similarity can be cosine similarity.

[0131] The first score is s′ LR (x, y), the second score is s′ RL (x, y).

[0132] The first weight acquisition method is:

[0133]

[0134] Among them, W L (x, y) is the first weight, and i, j represent the view traversal position.

[0135] The second weight acquisition method is:

[0136]

[0137] Among them, W R (x, y) is the second weight.

[0138] Substitute the preset first parameter and cosine similarity into a preset formula to obtain a first score, substitute the preset second parameter and cosine similarity into another preset formula to obtain a second score. According to the first score, a first weight is obtained, and according to the second score, a second weight is obtained.

[0139] The advantage of this setting is that through precise score calculation and weight distribution, efficient disparity fusion feature extraction is achieved, which significantly improves the effect and quality of video processing.

[0140] S103, determining a quality score of the video to be processed according to the parallax fusion feature and the preset text description; wherein the preset text description represents an evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0141] The evaluation dimension may refer to the problem type, problem location, severity, etc. of the problem that occurs in the video. Problem types may include: user experience, content rationality, visual coherence, etc.

[0142] In the training phase, the preset text description may include a description of the evaluation dimensions of the video to be processed, as well as a subjective score and evaluation description of the video to be processed, etc. The subjective score is obtained by experts scoring the video to be processed. In the testing phase, the text content only contains a description of the evaluation dimensions of the video to be processed. The subjective score and evaluation description of the video to be processed can be the label information used for model training.

[0143] In the embodiment of the present application, the quality score of the video to be processed is determined according to the parallax fusion feature and the preset text description, including:

[0144] The disparity fusion features and the preset text description are input into the preset large model to obtain the quality score of the output video to be processed.

[0145] In the training phase, the preset text description includes the type of stereoscopic video quality problem, the location of the problem, the severity, and the subjective score. For example, the preset text description can be: "This 3D video has excellent picture quality, bright colors, and rich details. The 3D effect is natural, and objects have a clear sense of depth and three-dimensionality, but there are some errors in the disparity map of some dynamic scenes, resulting in a slight flaw in the 3D effect; score: 7.5 / 10; how is the depth effect of this 3D video?". In the testing phase, the preset text description only includes the type of stereoscopic video quality problem, such as: Will this 3D video cause visual discomfort or fatigue?

[0146] During the training phase, the disparity fusion features and the preset text description are fused by splicing or other fusion mechanisms to obtain the fusion result of the disparity fusion features and the preset text description, and the fusion result is input into the preset large model to obtain the output quality score of the video to be processed and the evaluation text description.

[0147] The preset model may be a large language model.

[0148] During the testing phase, the preset text description is input into the preset large model to obtain the quality score of the video to be processed and the text description of the evaluation. For example: the input text description is: Will this 3D video cause visual discomfort or fatigue? The output is: "The 3D effect of this 3D video may cause visual discomfort or fatigue to some viewers. The main reason is that there are some mismatches in the disparity map and the disparity range is too large. This may cause visual illusions to the audience, such as incorrect occlusion relationships and too abrupt 3D effects; Rating: 5.5 / 10".

[0149] The advantage of this setting is that this method achieves efficient, accurate and automatic video quality assessment by leveraging the capabilities of large models, providing an important reference for video processing and optimization.

[0150] In the embodiment of the present application, it also includes:

[0151] Obtain a data set to be trained; wherein the data set to be trained includes a video to be trained, label data of the video to be trained, the label data representing an actual quality score of the video to be trained, and a text description to be trained;

[0152] The initial model is trained according to the data set to be trained to obtain a trained video scoring model; wherein the video scoring model is used to determine the left view data and the right view data of the video to be processed, and the disparity fusion features of the video to be processed are determined according to the left view data and the right view data; the quality score of the video to be processed is determined according to the disparity fusion features and a preset text description.

[0153] The label data of the video to be trained includes the problem type, problem location, and severity. Problem types include: user experience, content rationality, visual coherence, and actual quality score. The actual quality score is the score scored by expert review.

[0154] According to the training data set, the disparity fusion features of the video to be processed are obtained, and the disparity fusion features of the video to be processed are mapped to a space that is more suitable for processing by the large language model (i.e., the initial model). The label data of the video to be trained and the adaptation layer of the text description to be trained are used to optimize the mapping effect. The visual feature encoding is mapped to a unified feature space to facilitate the fusion of cross-modal information (i.e., the image features and text descriptions and quality scores of the video to be trained are fused), so that the image features can better adapt to the space processed by the subsequent large language model. The feature adaptation layer can be a fully connected layer, a self-attention mechanism, or a specially designed conversion network structure. Specifically, the input training data set is: question: "What is the depth effect of this 3D video?" and the label data of the video to be trained: "The depth effect of this 3D video is very realistic, with a strong sense of stereo, which brings an immersive viewing experience to the audience." The possible output of the model is: "The depth effect of this 3D video is very realistic, providing a significant sense of stereo, creating an immersive experience for the audience. During the training process, the output of the model goes back to try to match or approximate the label data of the manually annotated video to be trained.

[0155] The advantage of this setting is that through a systematic training process and data utilization, an efficient and accurate video scoring model is built, providing strong technical support for video quality assessment.

[0156] In the embodiment of the present application, the initial model is trained according to the data set to be trained to obtain a trained video scoring model, including:

[0157] Input the video to be trained into the initial model to obtain the fused feature vector of the video to be trained;

[0158] Converting the fused feature vector of the video to be trained into one-dimensional data, and determining the score difference between the one-dimensional data and the actual quality score of the video to be trained;

[0159] Determine the left view data and the right view data of the video to be trained, and obtain a first disparity loss value and a second disparity loss value according to the left view data and the right view data of the video to be trained; wherein the first disparity loss value represents the disparity of the left view data of the video to be trained relative to the right view data, and the second disparity loss value represents the disparity of the right view data of the video to be trained relative to the left view data;

[0160] Determine a difference between the first parallax loss value and the second parallax loss value as a loss difference;

[0161] Determine the feature difference between the fused feature vectors of adjacent frames in the video to be trained; wherein the feature difference represents the change between the fused feature vectors of adjacent frames;

[0162] If the score difference, loss difference, and feature difference all meet the preset difference conditions, it is determined that the initial model training is completed and the video scoring model is obtained.

[0163] The first disparity loss value is a disparity map from the left image to the right image.

[0164] The second disparity loss value is the disparity map from the right image to the left image. The left-to-right disparity and the right-to-left disparity essentially describe the same physical phenomenon, that is, the position difference when observing the same object from different angles. For high-quality 3D video, in theory, the right-to-left disparity is equal to the left-to-right disparity in terms of value, but in opposite directions. If the loss difference between the right-to-left disparity and the left-to-right disparity is less than a preset threshold (usually a pixel or less), the disparity of the two pixels is considered to be consistent, otherwise, they are considered to be inconsistent.

[0165] The quality score is the user's manual rating of the video to be trained.

[0166] Map the fused feature vector to a 1D vector, and obtain the score difference between the 1D vector and the actual quality score of the video to be trained. Calculate the disparity consistency loss between the left and right feature views. This is achieved by calculating the difference between the disparity map from the left image to the right image and the disparity map from the right image to the left image. Specifically, a simple method is to directly subtract the corresponding positions of the disparity map from the left image to the right image and the disparity map from the right image to the left image to obtain the difference. This compares the difference in the fused feature map vector Ff between adjacent frames to obtain the feature difference between the fused feature vectors of adjacent frames. For example, L1 loss can be used to measure the change of feature maps between adjacent frames, and then the change of feature maps between all adjacent frames is obtained, so that the overall change between all adjacent frames is small. If the score difference, loss difference, and feature difference (the change of feature maps between all adjacent frames is small) are all small, it is determined that the initial model training is completed. If the score difference, loss difference, and feature difference are arbitrarily large, reduce the learning rate and increase the batch size to train the model to be trained. When the score difference, loss difference, and feature difference between the one-dimensional data and the actual quality score of the video to be trained are all small, the initial model training is determined to be completed. The objective image features (i.e., the evaluation between adjacent frames represented by the loss difference, and the difference between the disparity map from the left image to the right image and the disparity map from the right image to the left image represented by the feature difference) are combined with the subjective evaluation (i.e., the user's subjective rating is represented by the score difference), so that the obtained model can evaluate the video quality more comprehensively and accurately.

[0167] The benefit of this setting is that through multi-dimensional feature analysis and optimization, the performance of the video scoring model is significantly improved, enabling it to more accurately and comprehensively evaluate video quality.

[0168] The present application provides a method for processing video data based on left and right views, which obtains the disparity fusion features of the video to be processed through the image frame of the left eye view and the right view data as the image frame of the right eye view, so that the video evaluation is more accurate and comprehensive. At the same time, by connecting the difference between the fusion feature vector and the quality score, the disparity between the left view data of the video to be trained relative to the right view data, the disparity between the right view data of the video to be trained relative to the left view data, and the feature difference between the fusion feature vectors of adjacent frames, in addition to the quality of the evaluation image itself, a subjective score evaluation is also added to make the video evaluation more comprehensive.

[0169] Figure 2 Another method for processing video data based on left and right views is provided for the present application. Figure 2 As shown, the execution subject of the method can be a server or other servers, and this embodiment is not particularly limited here. Figure 2 As shown, the method includes:

[0170] S201: Receive a left-view image sequence, a right-view image sequence, and text content of a stereoscopic video.

[0171] Among them, the objective quality score is the subjective scoring of the stereoscopic video by relevant experts.

[0172] During the training phase, the text content includes the type of stereoscopic video quality problem, the location of the problem, the severity, and the subjective rating. Specifically, the picture quality of this 3D video is excellent, with bright colors and rich details. The 3D effect is natural, and objects have a clear sense of depth and three-dimensionality, but there are some errors in the disparity map of some dynamic scenes, resulting in a slight flaw in the 3D effect. Rating: 7.5 / 10; During the testing phase (i.e. the user use phase), the text content only includes the type of stereoscopic video quality problem. Specifically, whether this 3D video will cause visual discomfort or fatigue.

[0173] During the training phase, a large set of stereo video datasets are collected, including left and right view image sequences, corresponding text descriptions, and objective quality scores.

[0174] (2) Perform preprocessing operations such as format unification, resolution adjustment, and frame synchronization on the left and right view image sequences.

[0175] (3) Clean and annotate the text description, including the type, location, severity and subjective score of the quality problem.

[0176] (4) Using the trained stereo matching algorithm (RAFT-Stereo), calculate the disparity image of each corresponding left and right view image in the sequence. Post-process the disparity image, for example, post-processing refers to using median filtering to reduce noise.

[0177] S202 : Using a stereo matching algorithm, calculate the left view image sequence and the right view image sequence to obtain a disparity image sequence.

[0178] Among them, the stereo matching algorithms include: RAFT-Stereo.

[0179] S203 , using a visual encoder, feature encoding the left view image sequence, the right view image sequence, and the disparity image sequence to obtain a left image feature encoding sequence, a right image feature encoding sequence, and a disparity image feature encoding sequence.

[0180] The pre-trained visual encoder is used to extract features from the left and right view image sequences and the disparity image sequence to obtain feature encoding. The visual encoder can use a Vision Transformer Encoder or an encoder improved based on a CNN (Convolutional Neural Network) to capture the spatial context information and local detail features of the image.

[0181] S204 , aligning features of the left image feature coding sequence, the right image feature coding sequence, and the disparity map feature coding sequence to ensure that they are in the same coordinate system.

[0182] Among them, a spatial transformer network (STN) is used to ensure that all features are encoded in the same coordinate system. STN can automatically learn how to adjust the position of the feature map to achieve better alignment.

[0183] S205, dynamically fuse the aligned left image feature coding sequence, the right image feature coding sequence and the disparity map feature coding sequence to obtain an attention weight matrix.

[0184] In the fusion process, the parallax consistency constraint is introduced, and the attention mechanism is used to dynamically adjust the weights of features from different sources, so that the fused features can reflect spatial information, temporal information and depth information at the same time.

[0185] The method of dynamically fusing the aligned left image feature coding sequence, the right image feature coding sequence and the disparity map feature coding sequence to obtain a more comprehensive and rich feature representation is as follows:

[0186] (1) Aligned feature map F L (Left figure feature coding sequence), F R (Characteristic coding sequence on the right), and F D (Disparity map feature encoding sequence).

[0187] (2) For each corresponding position (x, y) of the feature map, calculate the disparity consistency score s(x, y) between the features of the left and right images, which can be cosine similarity or other suitable metrics. At the same time, for the disparity map FD, obtain the disparity feature value d(x, y) at that position.

[0188] The cosine similarity is:

[0189]

[0190] Where (x, y) represents any position, F L Indicates the alignment vector of the left view data, F R represents the alignment vector of the right view data, and S(x, y) is the cosine similarity.

[0191] The consistency scores are:

[0192] s(x, y) = 1 - S(x, y)

[0193] (3) Generate an attention weight matrix based on the consistency score and disparity value.

[0194] α LR (x, y)=exp(-λ·|d(x, y)|)

[0195] α RL (x, y)=exp(-μ·|d(x, y)|)

[0196] Among them, d(x, y) is the alignment vector of the disparity map data at position (x, y), α LR , α RL are different directional weights, which can be preset constants or parameters learned through the network. λ and μ are adjustment parameters that control the weights to change with the disparity value.

[0197] s′ LR (x, y) = s(x, y)·α LR (x, y)

[0198] Among them, s′ LR (x, y) is the consistency score of the alignment vector of the left view data.

[0199] s′ RL (x, y) = s(x, y)·α RL (x, y)

[0200] Among them, s′ RL (x, y) is the consistency score of the alignment vector of the right view data.

[0201] The left view attention weight is:

[0202]

[0203] Among them, W L (x, y) is the left view attention weight, i, j represents the view traversal position.

[0204] The method for obtaining the right view attention weight is:

[0205]

[0206] Among them, W R (x, y) is the right view attention weight.

[0207] The method for obtaining the attention weight of the disparity map is:

[0208]

[0209] Among them, W D (x, y) is the disparity map attention weight.

[0210] The attention weight matrix is:

[0211] F f (x, y) = W L (x, y)×F L (x,y)+W R (x, y)×F R (x,y)+W D (x, y)×F D (x, y)

[0212] Among them, F L (x, y) is the feature coding sequence of the left image after alignment, F R (x, y) is the aligned right view encoding sequence, F D (x, y) is the aligned disparity map feature encoding sequence.

[0213] During training, the attention weight matrix Ff is mapped to a 1-dimensional vector and subtracted from the objective quality score to obtain the regression loss.

[0214] Calculate the left and right feature views F L , F R This is achieved by calculating the difference between the disparity map from the left image to the right image and the disparity map from the right image to the left image, that is, the difference between the left-to-right disparity and the right-to-left disparity.

[0215] Calculate the spatiotemporal consistency loss. This is achieved by comparing the difference between the feature maps Ff between adjacent frames. For example, L1 loss can be used to measure the change of feature maps between adjacent frames, so that the overall frame change is relatively stable.

[0216] When the regression loss, disparity consistency loss, and spatiotemporal consistency loss are all small, the training is completed.

[0217] S206: Input the attention weight matrix into the feature adaptation layer for visual feature mapping to obtain the mapped feature encoding.

[0218] S207: Encode the text feature of the text description about the video quality.

[0219] During the training phase, the text content includes the type of stereoscopic video quality problem, problem location, severity, subjective score, etc.; during the testing phase, the text content only includes the type of stereoscopic video quality problem.

[0220] Use a natural language processing model (such as BERT, GPT, etc.) to encode the input text description, extract semantic features in the text, and generate text feature encoding.

[0221] The visual feature encoding and text feature encoding are combined and input into the large language model.

[0222] Use labeled data to fine-tune the large language model so that it can understand and integrate visual and textual information and output accurate video quality evaluation.

[0223] S208, combining the attention weight matrix with the text feature encoding, and inputting it into a large language model through splicing or a more complex fusion mechanism, and outputting a comprehensive evaluation of the video quality. The large language model is trained to understand and integrate visual and text information.

[0224] During the testing phase (i.e., the user usage phase), the user inputs a stereoscopic video and a text description of the quality issues raised about the video. The trained model outputs an overall evaluation of the quality of the stereoscopic video based on the input visual and text information, including the type of problems that exist, the location of the problems, the severity, and the overall score. Specifically, the user inputs: stereoscopic video, and the question: Will this 3D video cause visual discomfort or fatigue? The answer is: The 3D effect of this 3D video may cause visual discomfort or fatigue to some viewers, mainly because there are some mismatches in the disparity map and the disparity range is too large. This may cause visual illusions to the audience, such as incorrect occlusion relationships and overly abrupt 3D effects. Rating: 5.5 / 10.

[0225] This application provides another method for processing video data based on left and right views, and provides a method for evaluating the quality of stereoscopic video based on a multimodal large model. By combining stereoscopic video features and a multimodal large model, the quality of stereoscopic video can be comprehensively and carefully evaluated. By extracting key stereoscopic features from the left and right view sequences and disparity map sequences of stereoscopic video, and performing feature alignment and disparity consistency fusion, a feature encoding that contains both detailed information and a global perspective is constructed.

[0226] In one possible implementation, you can also use Figure 3 represents another method for processing video data based on left and right views, such as Figure 3As shown, a left view sequence and a right view sequence are obtained according to a stereoscopic video, and a disparity map sequence (disparity sequence) is obtained according to the left view sequence (left sequence) and the right view sequence (right sequence), and then the left view sequence, the right view sequence, and the disparity map sequence are input into a visual encoder to obtain a left view sequence encoding, a right view sequence encoding, and a disparity map sequence encoding, and the left view sequence encoding, the right view sequence encoding, and the disparity map sequence encoding are aligned, and then the aligned left view sequence encoding, the aligned right view sequence encoding, and the aligned disparity map sequence encoding are dynamically fused for disparity consistency to obtain an attention weight matrix, and then the attention weight matrix is ​​input into a feature adaptation layer, and text content is input at the same time, and feature encoding is performed on the text content to obtain a text feature encoding, and in the feature adaptation layer, the attention weight matrix and the text feature are fused, and the attention weight matrix and the text feature obtained by the fusion processing are input into a large language model to obtain a video quality evaluation.

[0227] Figure 4 FIG. 1 is a diagram showing an example structure of a device for processing video data based on left and right views provided in an embodiment of the present application. Figure 4 As shown, the processing device 40 based on the video data of the left and right views includes: a first determination module 401, a second determination module 402, and a third determination module 403. Among them:

[0228] A first determination module 401 is used to determine left view data and right view data of a video to be processed; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view;

[0229] A second determination module 402 is used to determine the disparity fusion feature of the video to be processed according to the left view data and the right view data; wherein the disparity fusion feature represents the left and right eye views and the disparity map of the left and right eyes;

[0230] The third determination module 403 is used to determine the quality score of the video to be processed according to the parallax fusion feature and the preset text description; wherein the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0231] Figure 5 This is an example diagram of the structure of a processing device for video data based on left and right views provided in an embodiment of the present application. Figure 5 As shown, the processing device 50 based on the video data of the left and right views includes: a first determination module 501, a second determination module 502, and a third determination module 503, the second determination module 502 includes an obtaining unit 5021 and a determining unit 5022, and the third determination module 503 includes an obtaining unit 5031, wherein:

[0232] In one example, the second determining module 502 includes:

[0233] An obtaining unit 5021 is used to obtain disparity map data according to the left view data and the right view data;

[0234] The determination unit 5022 is used to determine the disparity fusion features of the video to be processed according to the left view data, the right view data, and the disparity map data.

[0235] In a possible implementation, unit 5021 is obtained, which is specifically used for:

[0236] Encoding the left view data, the right view data, and the disparity data respectively to obtain a feature vector of the left view data, a feature vector of the right view data, and a feature vector of the disparity data;

[0237] The disparity fusion feature of the video to be processed is obtained according to the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data.

[0238] In a possible implementation, unit 5021 is obtained, which is specifically used for:

[0239] Aligning the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data to obtain an alignment vector of the left view data, an alignment vector of the right view data, and an alignment vector of the disparity map data; wherein the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data are located in the same coordinate system;

[0240] The disparity fusion feature of the video to be processed is obtained according to the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data.

[0241] In a possible implementation, unit 5021 is obtained, which is specifically used for:

[0242] Determining a similarity between an alignment vector of the left view data and an alignment vector of the right view data;

[0243] According to the similarity, determine the weight of the left view data as a first weight, and determine the weight of the right view data as a second weight;

[0244] Determine the weight of the disparity map according to the alignment vector of the disparity map data, which is the third weight;

[0245] A disparity fusion feature of the video to be processed is determined according to an alignment vector of the left view data, an alignment vector of the right view data, an alignment vector of the disparity map data, a first weight, a second weight, and a third weight.

[0246] In a possible implementation, unit 5021 is obtained, which is specifically used for:

[0247] Obtaining a first score according to the similarity and a preset first parameter; wherein the first score represents a consistency score of the left view data relative to the right view data;

[0248] Obtaining a second score according to the similarity and a preset second parameter; wherein the second score represents a consistency score of the right view data relative to the left view data;

[0249] According to the first score, the weight of the left view data is obtained, which is the first weight, and according to the second score, the weight of the right view data is determined, which is the second weight.

[0250] In one example, the third determining module 503 includes:

[0251] The obtaining unit 5301 is used to input the disparity fusion feature and the preset text description into the preset large model to obtain the quality score of the output video to be processed.

[0252] In a possible implementation, unit 5301 is obtained, which is specifically used for:

[0253] Obtain a data set to be trained; wherein the data set to be trained includes a video to be trained, label data of the video to be trained, the label data representing an actual quality score of the video to be trained, and a text description to be trained;

[0254] The initial model is trained according to the data set to be trained to obtain a trained video scoring model; wherein the video scoring model is used to determine the left view data and the right view data of the video to be processed, and the disparity fusion features of the video to be processed are determined according to the left view data and the right view data; the quality score of the video to be processed is determined according to the disparity fusion features and a preset text description.

[0255] In a possible implementation, unit 5301 is obtained, which is specifically used for:

[0256] Input the video to be trained into the initial model to obtain the fused feature vector of the video to be trained;

[0257] Converting the fused feature vector of the video to be trained into one-dimensional data, and determining the score difference between the one-dimensional data and the actual quality score of the video to be trained;

[0258] Determine the left view data and the right view data of the video to be trained, and obtain a first disparity loss value and a second disparity loss value according to the left view data and the right view data of the video to be trained; wherein the first disparity loss value represents the disparity of the left view data of the video to be trained relative to the right view data, and the second disparity loss value represents the disparity of the right view data of the video to be trained relative to the left view data;

[0259] Determine a difference between the first parallax loss value and the second parallax loss value as a loss difference;

[0260] Determine the feature difference between the fused feature vectors of adjacent frames in the video to be trained; wherein the feature difference represents the change between the fused feature vectors of adjacent frames;

[0261] If the score difference, loss difference, and feature difference all meet the preset difference conditions, it is determined that the initial model training is completed and the video scoring model is obtained.

[0262] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 60 includes:

[0263] The electronic device 60 may include a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a communication component 603 and other components. The processor 601 , the memory 602 and the communication component 603 are connected via a bus 604 .

[0264] In a specific implementation process, at least one processor 601 executes the computer-executable instructions stored in the memory 602, so that the at least one processor 601 executes the above-mentioned method for processing video data based on left and right views.

[0265] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0266] In the above Figure 6In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0267] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0268] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0269] In some embodiments, a computer program product is further proposed, including a computer program or an instruction, which implements the steps in any of the above-mentioned methods for processing video data based on left and right views when executed by a processor.

[0270] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0271] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0272] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions can be loaded by a processor to execute the steps in any one of the processing methods for video data based on left and right views provided in the embodiment of the present application.

[0273] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0274] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.

[0275] Since the instructions stored in the storage medium can execute the steps in any one of the methods for processing video data based on left and right views provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the methods for processing video data based on left and right views provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0276] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0277] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0278] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0279] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0280] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0281] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0282] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0283] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0284] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0285] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0286] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for processing video data based on left and right views, characterized in that: The method comprises: Determine left view data and right view data of the video to be processed; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view; Determine the disparity fusion feature of the video to be processed according to the left view data and the right view data; wherein the disparity fusion feature represents the views of the left and right eyes and the disparity maps of the left and right eyes; The quality score of the video to be processed is determined according to the disparity fusion feature and the preset text description; wherein the preset text description represents an evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

2. The method according to claim 1, characterized in that The determining, according to the left view data and the right view data, the parallax fusion feature of the video to be processed comprises: Obtaining disparity map data according to the left view data and the right view data; The disparity fusion feature of the video to be processed is determined according to the left view data, the right view data, and the disparity map data.

3. The method according to claim 2, characterized in that The determining, according to the left view data, the right view data, and the disparity map data, a disparity fusion feature of the video to be processed includes: respectively encoding the left view data, the right view data, and the disparity data to obtain a feature vector of the left view data, a feature vector of the right view data, and a feature vector of the disparity data; The disparity fusion feature of the video to be processed is obtained according to the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data.

4. The method according to claim 3, characterized in that The obtaining the disparity fusion feature of the video to be processed according to the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data comprises: Performing alignment processing on the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data to obtain an alignment vector of the left view data, an alignment vector of the right view data, and an alignment vector of the disparity map data; wherein the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data are located in the same coordinate system; The disparity fusion feature of the video to be processed is obtained according to the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data.

5. The method according to claim 4, characterized in that The obtaining the disparity fusion feature of the video to be processed according to the alignment vector of the left view data, the alignment vector of the right view data, and the alignment vector of the disparity map data comprises: Determining a similarity between an alignment vector of the left view data and an alignment vector of the right view data; According to the similarity, determining a weight of the left view data as a first weight, and determining a weight of the right view data as a second weight; Determining a weight of the disparity map according to the alignment vector of the disparity map data, which is a third weight; The disparity fusion feature of the video to be processed is determined according to the alignment vector of the left view data, the alignment vector of the right view data, the alignment vector of the disparity map data, the first weight, the second weight, and the third weight.

6. The method according to claim 5, characterized in that The step of determining the weight of the left view data as a first weight and determining the weight of the right view data as a second weight according to the similarity includes: Obtaining a first score according to the similarity and a preset first parameter; wherein the first score represents a consistency score of the left view data relative to the right view data; Obtaining a second score according to the similarity and a preset second parameter; wherein the second score represents a consistency score of the right view data relative to the left view data; According to the first score, the weight of the left view data is obtained, which is a first weight, and according to the second score, the weight of the right view data is determined, which is a second weight.

7. The method according to claim 1, characterized in that Determining the quality score of the video to be processed according to the parallax fusion feature and the preset text description includes: The disparity fusion feature and the preset text description are input into a preset large model to obtain the output quality score of the video to be processed.

8. The method according to any one of claims 1 to 7, characterized in that Also includes: Acquire a data set to be trained; wherein the data set to be trained includes a video to be trained, label data of the video to be trained, the label data representing an actual quality score of the video to be trained, and a text description to be trained; The initial model is trained according to the data set to be trained to obtain a trained video scoring model; wherein the video scoring model is used to determine the left view data and the right view data of the video to be processed, and determine the disparity fusion features of the video to be processed according to the left view data and the right view data; and determine the quality score of the video to be processed according to the disparity fusion features and a preset text description.

9. The method according to claim 8, characterized in that The step of training the initial model according to the data set to be trained to obtain a trained video scoring model includes: Inputting the video to be trained into the initial model to obtain a fusion feature vector of the video to be trained; Converting the fused feature vector of the video to be trained into one-dimensional data, and determining a score difference between the one-dimensional data and an actual quality score of the video to be trained; Determine the left view data and the right view data of the video to be trained, and obtain a first disparity loss value and a second disparity loss value according to the left view data and the right view data of the video to be trained; wherein the first disparity loss value represents the disparity of the left view data of the video to be trained relative to the right view data, and the second disparity loss value represents the disparity of the right view data of the video to be trained relative to the left view data; determining a difference between the first parallax loss value and the second parallax loss value as a loss difference; Determine a feature difference between fused feature vectors of adjacent frames in the video to be trained; wherein the feature difference represents a change between fused feature vectors of adjacent frames; If the score difference, loss difference, and feature difference all meet the preset difference conditions, it is determined that the initial model training is completed and the video scoring model is obtained.

10. A device for processing video data based on left and right views, characterized in that: include: A first determination module is used to determine left view data and right view data of the video to be processed; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view; A second determination module is used to determine the disparity fusion feature of the video to be processed according to the left view data and the right view data; wherein the disparity fusion feature represents the views of the left and right eyes and the disparity maps of the left and right eyes; The third determination module is used to determine the quality score of the video to be processed according to the disparity fusion feature and the preset text description; wherein the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images