Method, apparatus and medium for processing video data based on left and right views

Abstract

The application provides a processing method and device of video data based on left and right views, equipment and medium, the method comprises: determining left view data and right view data of a to-be-processed video; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view; determining a parallax fusion feature of the to-be-processed video according to the left view data and the right view data; wherein the parallax fusion feature represents a left eye view and a left eye parallax map; determining a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description; wherein the preset text description represents an evaluation dimension of the to-be-processed video, and the quality score represents a video quality of the to-be-processed video. The method of the application determines the parallax fusion feature of the to-be-processed video through the left view data and the right view data, and the parallax fusion feature of the to-be-processed video can improve the accuracy of data processing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device and medium for processing video data based on left and right views. Background Technology

[0002] Stereoscopic video (i.e., 3D video) is widely used in fields such as virtual reality, mixed reality, and holographic projection. Its three-dimensional depth perception characteristics far exceed those of two-dimensional video, making it closer to the real human viewing experience.

[0003] However, stereoscopic video is susceptible to noise interference during construction, encoding, transmission, and display, which can affect video quality and cause visual discomfort. Therefore, reasonable evaluation of stereoscopic video is crucial for its improvement. Thus, accurate evaluation of stereoscopic video is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for processing video data based on left and right views, in order to solve the problem of accurately evaluating stereoscopic video.

[0005] In a first aspect, this application provides a method for processing video data based on left and right views, including:

[0006] Determine the left view data and right view data of the video to be processed; wherein, the left view data is the image frame of the left eye view, and the right view data is the image frame of the right eye view;

[0007] Based on the left and right view data, the disparity fusion features of the video to be processed are determined; whereby, the disparity fusion features represent the views of the left and right eyes and the disparity maps of the left and right eyes.

[0008] Based on the parallax fusion features and the preset text description, the quality score of the video to be processed is determined; whereby the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0009] Optionally, the method described above determines the disparity fusion features of the video to be processed based on the left and right view data, including:

[0010] Based on the left view data and the right view data, the disparity map data is obtained;

[0011] Based on the left view data, right view data, and disparity map data, the disparity fusion features of the video to be processed are determined.

[0012] Optionally, the method described above determines the disparity fusion features of the video to be processed based on the left view data, right view data, and disparity map data, including:

[0013] The left view data, right view data, and disparity data are encoded separately to obtain the feature vectors of the left view data, right view data, and disparity data.

[0014] Based on the feature vectors of the left view data, the right view data, and the disparity map data, the disparity fusion features of the video to be processed are obtained.

[0015] Optionally, as described above, the disparity fusion features of the video to be processed are obtained based on the feature vectors of the left view data, the right view data, and the disparity map data, including:

[0016] Alignment processing is performed on the feature vectors of the left view data, the right view data, and the disparity map data to obtain the alignment vectors of the left view data, the right view data, and the disparity map data; wherein the alignment vectors of the left view data, the right view data, and the disparity map data are located in the same coordinate system;

[0017] Based on the alignment vectors of the left view data, the right view data, and the disparity map data, the disparity fusion features of the video to be processed are obtained.

[0018] Optionally, as described above, the disparity fusion features of the video to be processed are obtained based on the alignment vectors of the left view data, the right view data, and the disparity map data, including:

[0019] Determine the similarity between the alignment vectors of the left view data and the alignment vectors of the right view data;

[0020] Based on similarity, the weight of the left view data is determined as the first weight, and the weight of the right view data is determined as the second weight;

[0021] The weights of the disparity map are determined based on the alignment vector of the disparity map data; these weights are designated as the third weight.

[0022] Based on the alignment vectors of the left view data, the right view data, the disparity map data, the first weight, the second weight, and the third weight, the disparity fusion features of the video to be processed are determined.

[0023] Optionally, the method described above determines the weight of the left view data (as the first weight) and the weight of the right view data (as the second weight) based on similarity, including:

[0024] A first score is obtained based on the similarity and a preset first parameter; whereby the first score represents the consistency score between the left view data and the right view data.

[0025] A second score is obtained based on the similarity and a preset second parameter; whereby the second score represents the consistency score of the right view data relative to the left view data.

[0026] Based on the first score, the weight of the left view data is obtained, which is the first weight; and based on the second score, the weight of the right view data is determined, which is the second weight.

[0027] Optionally, the method described above determines the quality score of the video to be processed based on parallax fusion features and a preset text description, including:

[0028] The parallax fusion features and the preset text description are input into the preset large model to obtain the quality score of the output video to be processed.

[0029] Alternatively, the method described above may also include:

[0030] Obtain the training dataset; the training dataset includes the training video, the label data of the training video, the label data representing the actual quality score of the training video, and the training text description;

[0031] The initial model is trained using the dataset to be trained to obtain a trained video scoring model. The video scoring model is used to determine the left and right view data of the video to be processed, and to determine the disparity fusion features of the video to be processed based on the left and right view data. The quality score of the video to be processed is determined based on the disparity fusion features and the preset text description.

[0032] Optionally, as described above, the initial model is trained using the dataset to be trained to obtain a trained video scoring model, including:

[0033] The video to be trained is input into the initial model to obtain the fused feature vector of the video to be trained;

[0034] The fused feature vector of the video to be trained is converted into one-dimensional data, and the score difference between the one-dimensional data and the actual quality score of the video to be trained is determined.

[0035] The left and right view data of the video to be trained are determined, and a first disparity loss value and a second disparity loss value are obtained based on the left and right view data of the video to be trained; wherein, the first disparity loss value represents the disparity of the left view data of the video to be trained relative to the right view data, and the second disparity loss value represents the disparity of the right view data of the video to be trained relative to the left view data.

[0036] The difference between the first disparity loss value and the second disparity loss value is determined as the loss difference.

[0037] Determine the feature difference between the fused feature vectors of adjacent frames in the video to be trained; where the feature difference represents the change between the fused feature vectors of adjacent frames.

[0038] If the score difference, loss difference, and feature difference all meet the preset difference conditions, then the initial model training is complete, and the video scoring model is obtained.

[0039] Secondly, this application provides a processing apparatus for video data based on left and right views, comprising:

[0040] The first determining module is used to determine the left view data and right view data of the video to be processed; wherein, the left view data is the image frame of the left eye view and the right view data is the image frame of the right eye view;

[0041] The second determining module is used to determine the disparity fusion features of the video to be processed based on the left view data and the right view data; wherein, the disparity fusion features represent the views of the left and right eyes and the disparity maps of the left and right eyes.

[0042] The third determining module is used to determine the quality score of the video to be processed based on the parallax fusion features and the preset text description; wherein, the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0043] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0044] The memory stores instructions that the computer executes;

[0045] The processor executes computer execution instructions stored in memory to implement the method in this application.

[0046] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of this application.

[0047] Fifthly, this application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0048] This application provides a method, apparatus, device, and medium for processing video data based on left and right views. The method determines the left and right view data of the video to be processed; wherein the left view data consists of image frames from the left eye view, and the right view data consists of image frames from the right eye view. Based on the left and right view data, the method determines the disparity fusion features of the video to be processed; wherein the disparity fusion features represent the views of the left and right eyes and the disparity maps of the left and right eyes. By obtaining the fusion features from the left and right view data, a comprehensive evaluation can be obtained to address quality issues raised in stereoscopic videos. Based on the disparity fusion features and a preset text description, a quality score is determined for the video to be processed. By considering multiple data points, a more accurate, detailed, and comprehensive evaluation can be obtained, resulting in a quality score for the video to be processed. The preset text description represents the evaluation dimensions of the video to be processed, and the quality score represents the video quality of the video to be processed. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 A flowchart illustrating a method for processing video data based on left and right views, provided in an embodiment of this application;

[0051] Figure 2 A flowchart illustrating another method for processing video data based on left and right views, provided in an embodiment of this application;

[0052] Figure 3 A flowchart illustrating another method for processing video data based on left and right views, provided in an embodiment of this application;

[0053] Figure 4 A schematic diagram of the structure of a video data processing device based on left and right views provided in an embodiment of this application;

[0054] Figure 5 A schematic diagram of the structural architecture of a video data processing device based on left and right views provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the electronic device structure provided in an embodiment of this application.

[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0058] Currently, existing evaluations of stereoscopic videos primarily rely on extracting objective features from the images and attempting to fit or approximate subjective human ratings using these features. However, these methods are limited by their single-dimensional analytical framework, focusing on improving one or a few image features. Since the quality of stereoscopic videos is influenced by many complex factors, these methods often fail to comprehensively and accurately reflect the true quality of stereoscopic videos.

[0059] The video data processing method based on left and right views provided in this application aims to solve the above-mentioned technical problems of the prior art.

[0060] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0061] This application provides a method, apparatus, device, and medium for processing video data based on left and right views. The method determines the left and right view data of the video to be processed; wherein the left view data consists of image frames from the left eye view, and the right view data consists of image frames from the right eye view. Based on the left and right view data, the method determines the disparity fusion features of the video to be processed; wherein the disparity fusion features represent the views of the left and right eyes and the disparity maps of the left and right eyes. By obtaining the fusion features from the left and right view data, a comprehensive evaluation can be obtained to address quality issues raised in stereoscopic videos. Based on the disparity fusion features and a preset text description, a quality score is determined for the video to be processed. By considering multiple data points, a more accurate, detailed, and comprehensive evaluation can be obtained, resulting in a quality score for the video to be processed. The preset text description represents the evaluation dimensions of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0062] Figure 1 This is a flowchart illustrating a method for processing video data based on left and right views, provided as an embodiment of this application. Figure 1 As shown, the execution subject of this method can be a server or other servers; this embodiment does not impose any special restrictions here. Figure 1As shown, the method includes:

[0063] S101. Determine the left view data and right view data of the video to be processed; wherein, the left view data is the image frame of the left eye view and the right view data is the image frame of the right eye view.

[0064] The left view refers to the view obtained by observing with the left eye.

[0065] The right view refers to the view obtained by observing with the right eye.

[0066] The video to be processed is a 3D video. Specific frames are extracted from the video stream using FBS (Frame Blending Sampling Technique) to obtain left-view and right-view data. The extracted frames can be equally spaced or based on specific events or changes (such as scene transitions).

[0067] S102. Based on the left view data and the right view data, determine the disparity fusion features of the video to be processed; wherein, the disparity fusion features represent the views of the left and right eyes and the disparity maps of the left and right eyes.

[0068] Specifically, a disparity map is obtained based on the left and right view data. Then, the left and right views and the disparity map are fused to obtain the disparity fusion features of the video to be processed.

[0069] For example, a disparity map is obtained by combining left view data, right view data, and a preset disparity calculation algorithm (RAFT-Stereo). The preset disparity algorithm can be a stereo matching algorithm (RAFT-Stereo).

[0070] The fusion process can assign different weights to different left image feature coding sequences, right image feature coding sequences, and disparity map feature coding sequences, and then obtain the disparity fusion features of the video to be processed by weighted summation.

[0071] In this embodiment of the application, the disparity fusion features of the video to be processed are determined based on the left view data and the right view data, including:

[0072] Based on the left view data and the right view data, the disparity map data is obtained;

[0073] Based on the left view data, right view data, and disparity map data, the disparity fusion features of the video to be processed are determined.

[0074] Parallax maps are an image processing technique used to calculate and represent depth information, commonly found in computer vision, 3D graphics rendering, and virtual reality. By recording depth information at different points in a scene, parallax maps help generate images with a sense of depth and stereoscopic effect. A parallax map represents the displacement of each pixel between the left and right views.

[0075] To obtain disparity data using left and right view data, it's essential to first ensure that the left and right view data are on the same horizontal line. Then, a matching cost is calculated for each pixel in both the left and right view data. This matching cost is used to compare the similarity of pixels in the left and right views. Finally, the best match is found within the disparity range based on the similarity of pixels in the left and right views. For example, for block matching: a small block (usually a rectangular region) is selected in one image (e.g., the left view), and then a matching block is searched in another image (e.g., the right view). The matching criteria are usually based on some similarity metric, such as the sum of absolute differences, the sum of squared differences, or normalized cross-correlation. The "StereoBM" class provided by OpenCV (Open Source Computer Vision Library) can be used for block matching to obtain disparity map data. Post-processing of the disparity map data (i.e., smoothing and noise removal) yields a higher-quality disparity map. Alternatively, disparity map data can be obtained using stereo matching algorithms, left view data, and right view data.

[0076] The disparity map data of the left and right view data is calculated by the stereo matching algorithm (RAFT-Stereo). Then, different weights are assigned to the left view data, right view data, and disparity map data. The weighted sum is then performed to obtain the disparity fusion features of the video to be processed. The different weights can be calculated by a preset formula or preset by the user.

[0077] The advantage of this setup is that by generating a disparity map, depth information in the scene can be captured more accurately. This makes depth perception more realistic and natural when the video is viewed on a 3D display device. The disparity fusion feature takes into account left view data, right view data, and disparity map data, resulting in a more accurate evaluation.

[0078] In this embodiment of the application, the disparity fusion features of the video to be processed are determined based on the left view data, right view data, and disparity map data, including:

[0079] The left view data, right view data, and disparity data are encoded separately to obtain the feature vectors of the left view data, right view data, and disparity data.

[0080] Based on the feature vectors of the left view data, the right view data, and the disparity map data, the disparity fusion features of the video to be processed are obtained.

[0081] Encoding refers to feature encoding, a common technique in data preprocessing, especially in machine learning and data mining. Its purpose is to transform raw data into a numerical format suitable for machine learning algorithms. Feature encoding is particularly important for processing non-numerical data (such as categorical data).

[0082] In machine learning, a feature vector typically refers to a vector used to describe an instance of data. Each element represents a feature or attribute. For example, in image processing, a feature vector might contain pixel values, a color histogram, or other image features.

[0083] A visual encoder is used to encode the features of the left view data, right view data, and disparity data respectively, resulting in a left image feature encoding sequence (i.e., the feature vector of the left view data), a right image feature encoding sequence (i.e., the feature vector of the right view data), and a disparity map feature encoding sequence (i.e., the feature vector of the disparity map data). The left image feature encoding sequence, right image feature encoding sequence, and disparity map feature encoding sequence are then fused. The fusion process can assign different weights to different left image feature encoding sequences, right image feature encoding sequences, and disparity map feature encoding sequences. Finally, the disparity fusion features of the video to be processed are obtained by weighted summation.

[0084] The advantages of this setup are as follows: Through encoding processing, the left view data, right view data, and disparity map data can be better integrated. By fusing the features of the left view, right view, and disparity map, the depth information of the scene can be captured more accurately, thereby improving the accuracy of depth perception. Feature fusion can effectively reduce noise and errors that may exist in a single view or disparity map, improving the reliability of the overall data.

[0085] In this embodiment of the application, the disparity fusion features of the video to be processed are obtained based on the feature vectors of the left view data, the right view data, and the disparity map data, including:

[0086] Alignment processing is performed on the feature vectors of the left view data, the right view data, and the disparity map data to obtain the alignment vectors of the left view data, the right view data, and the disparity map data; wherein the alignment vectors of the left view data, the right view data, and the disparity map data are located in the same coordinate system;

[0087] Based on the alignment vectors of the left view data, the right view data, and the disparity map data, the disparity fusion features of the video to be processed are obtained.

[0088] Alignment processing typically refers to selecting the feature vectors of the disparity map data as the reference coordinate system. This means transforming the feature vectors of the left view data and the right view data into the disparity feature coordinate system. Specifically, assuming the feature vectors of the disparity map data are in the range of [-1, 1] and the feature vectors of the left view data are in the range of [0, 255], unifying them into the same coordinate system means transforming the dimensions of the feature vector values ​​of the left view data to the range of [-1, 1].

[0089] The feature vectors of the disparity map data can be selected as the reference coordinate system. The feature vectors of the left view data and the right view data are transformed into the disparity feature coordinate system, thereby obtaining the alignment vectors of the left view data, the right view data, and the disparity map data. The alignment vectors of the left view data, the right view data, and the disparity map data are then fused to obtain the disparity fusion features of the video to be processed.

[0090] The advantage of this setup is that by aligning data from different views to the same coordinate system, feature inconsistencies caused by perspective differences are reduced, thereby improving the accuracy of feature fusion. Alignment reduces uncertainties caused by perspective changes and parallax errors, making subsequent video processing more robust and stable.

[0091] In this embodiment of the application, the disparity fusion features of the video to be processed are obtained based on the alignment vectors of the left view data, the right view data, and the disparity map data, including:

[0092] Determine the similarity between the alignment vectors of the left view data and the alignment vectors of the right view data;

[0093] Based on similarity, the weight of the left view data is determined as the first weight, and the weight of the right view data is determined as the second weight;

[0094] The weights of the disparity map are determined based on the alignment vector of the disparity map data; these weights are designated as the third weight.

[0095] Based on the alignment vectors of the left view data, the right view data, the disparity map data, the first weight, the second weight, and the third weight, the disparity fusion features of the video to be processed are determined.

[0096] Here, similarity can refer to cosine similarity or cosine distance. Cosine distance and cosine similarity are complementary, and are calculated as 1 minus cosine similarity.

[0097] The cosine similarity is:

[0098]

[0099] Where (x, y) represents any position, F L F represents the alignment vector of the left view data. R Let S(x, y) represent the alignment vector of the right view data, and let S(x, y) be the cosine similarity.

[0100] The cosine distance is:

[0101] s(x, y) = 1 - S(x, y)

[0102] Where s(x, y) is the cosine distance.

[0103] α LR (x, y)=exp(-λ·|d(x, y)|)

[0104] α RL (x, y)=exp(-μ·|d(x, y)|)

[0105] Where d(x, y) is the alignment vector of the disparity map data at position (x, y), and α LR α RL The weights are for different directions and can be pre-set constants or parameters learned by the network. λ and μ are adjustment parameters that control the weights as they change with the disparity value.

[0106] s′ LR (x, y) = s(x, y)·α LR (x, y)

[0107] Among them, s′ LR (x, y) represents the consistency score of the left view data alignment vector.

[0108] s′ RL (x, y) = s(x, y)·α RL (x, y)

[0109] Among them, s′ RL (x, y) represents the consistency score of the right view data alignment vector.

[0110] The method for obtaining the first weight is as follows:

[0111]

[0112] Among them, W L (x, y) is the first weight, and i, j represent the view traversal positions.

[0113] The method for obtaining the second weight is as follows:

[0114]

[0115] Among them, W R (x, y) is the second weight.

[0116] The method for obtaining the third weight is as follows:

[0117]

[0118] Among them, W D (x, y) is the third weight.

[0119] The parallax fusion features of the video to be processed are:

[0120] F f (x, y) = W L (x, y) × F L (x, y) + W R (x, y) × F R (x, y) + W D (x, y) × F D (x, y)

[0121] Among them, F L (x, y) is the alignment vector of the left view data, F R (x, y) is the alignment vector of the right view.

[0122] F D (x, y)

[0123] This is the alignment vector for the disparity map data.

[0124] Confirm the cosine similarity between the alignment vectors of the left and right view data. Based on the cosine similarity, determine the cosine distance. Substitute the cosine distance into a preset formula to obtain the weights of the left and right view data. Based on the alignment vectors of the disparity map data, obtain the alignment vector of the disparity map. Combine the alignment vectors of the left and right view data, the alignment vectors of the disparity map data, the weights of the left and right view data, and the weights of the disparity map to obtain the disparity fusion features of the video to be processed.

[0125] The advantage of this setup is that it enables efficient parallax fusion feature extraction through precise similarity calculation and weight allocation, significantly improving the effect and quality of video processing.

[0126] In this embodiment of the application, determining the weight of the left view data as a first weight and determining the weight of the right view data as a second weight based on similarity includes:

[0127] A first score is obtained based on the similarity and a preset first parameter; whereby the first score represents the consistency score between the left view data and the right view data.

[0128] A second score is obtained based on the similarity and a preset second parameter; whereby the second score represents the consistency score of the right view data relative to the left view data.

[0129] Based on the first score, the weight of the left view data is obtained, which is the first weight; and based on the second score, the weight of the right view data is determined, which is the second weight.

[0130] The first parameter is preset to α. LR (x, y), with the second parameter preset to α. RL The weights for different directions can be pre-defined constants. The similarity can be cosine similarity.

[0131] The first score is s′ LR (x, y), the second score is s′ RL (x, y).

[0132] The method for obtaining the first weight is as follows:

[0133]

[0134] Among them, W L (x, y) is the first weight, and i, j represent the view traversal positions.

[0135] The method for obtaining the second weight is as follows:

[0136]

[0137] Among them, W R (x, y) is the second weight.

[0138] Substituting the preset first parameter and cosine similarity into a preset formula yields the first score. Substituting the preset second parameter and cosine similarity into another preset formula yields the second score. Based on the first score, the first weight is obtained; based on the second score, the second weight is obtained.

[0139] The advantage of this setup is that it enables efficient parallax fusion feature extraction through precise score calculation and weight allocation, significantly improving the effect and quality of video processing.

[0140] S103. Determine the quality score of the video to be processed based on the parallax fusion features and the preset text description; wherein, the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0141] The evaluation dimensions can refer to assessing the type, location, and severity of problems that appear in the video. Problem types can include: user experience, content rationality, and visual coherence.

[0142] During the training phase, the pre-defined text description can include descriptions of the evaluation dimensions of the video to be processed, as well as subjective scores and evaluations. Subjective scores are obtained through expert ratings of the video. During the testing phase, the text content only includes descriptions of the evaluation dimensions of the video to be processed. The subjective scores and evaluations can be label information used for model training.

[0143] In this embodiment of the application, the quality score of the video to be processed is determined based on parallax fusion features and a preset text description, including:

[0144] The parallax fusion features and the preset text description are input into the preset large model to obtain the quality score of the output video to be processed.

[0145] During the training phase, the pre-set text descriptions include the type of stereoscopic video quality issue, its location, severity, and subjective rating. For example, a pre-set text description might be: "This 3D video has excellent image quality, vibrant colors, and rich details. The 3D effect is natural, and objects have a clear sense of depth and stereoscopicity. However, there are some errors in the disparity map of some dynamic scenes, resulting in slight flaws in the 3D effect; rating: 7.5 / 10; how is the depth effect of this 3D video?" During the testing phase, the pre-set text descriptions only include the type of stereoscopic video quality issue, such as: Will this 3D video cause visual discomfort or fatigue?

[0146] During the training phase, the parallax fusion features and the preset text description are spliced ​​together or other fusion mechanisms to obtain the fusion result of the parallax fusion features and the preset text description. The fusion result is then input into the preset large model to obtain the quality score of the output video and the evaluation text description.

[0147] The preset model can be a large language model.

[0148] During the testing phase, a pre-defined text description is input into a pre-defined large model to obtain the quality score of the video to be processed and the evaluation text description. For example, if the input text description is: "Will this 3D video cause visual discomfort or fatigue?", the output would be: "The 3D effect of this 3D video may cause visual discomfort or fatigue for some viewers, mainly due to some mismatches in the disparity map and excessively large disparity ranges. This may cause viewers to experience visual illusions, such as incorrect occlusion relationships and overly abrupt 3D effects; Rating: 5.5 / 10."

[0149] The advantage of this setup is that, by leveraging the capabilities of large models, this method achieves efficient, accurate, and automated video quality assessment, providing important reference data for video processing and optimization.

[0150] This application embodiment also includes:

[0151] Obtain the training dataset; the training dataset includes the training video, the label data of the training video, the label data representing the actual quality score of the training video, and the training text description;

[0152] The initial model is trained using the dataset to be trained to obtain a trained video scoring model. The video scoring model is used to determine the left and right view data of the video to be processed, and to determine the disparity fusion features of the video to be processed based on the left and right view data. The quality score of the video to be processed is determined based on the disparity fusion features and the preset text description.

[0153] The tag data for the training videos includes problem type, problem location, and severity. Problem types include: user experience, content appropriateness, visual coherence, and actual quality score. The actual quality score is the score given by expert reviewers.

[0154] Based on the training dataset, the disparity fusion features of the videos to be processed are obtained, and these features are mapped to a space more suitable for processing by the large language model (i.e., the initial model). The mapping effect is optimized using the labeled data of the training videos and the text description adaptation layer. Visual features are encoded and mapped to a unified feature space to facilitate the fusion of cross-modal information (i.e., the fusion of graph features, text descriptions, and quality scores of the training videos), allowing image features to better adapt to the subsequent processing space of the large language model. The feature adaptation layer can be a fully connected layer, a self-attention mechanism, or a specially designed transformation network structure. Specifically, the input training dataset consists of the question: "How is the depth effect of this 3D video?", and the labeled data of the training videos: "The depth effect of this 3D video is very realistic, with a strong sense of stereoscopic effect, providing viewers with an immersive viewing experience." The possible output of the model is: "The depth effect of this 3D video is very realistic, providing a significant sense of stereoscopic effect, creating an immersive experience for viewers." During training, the model's output attempts to match or approximate the manually labeled data of the training videos.

[0155] The advantage of this setup is that, through a systematic training process and data utilization, an efficient and accurate video scoring model is built, providing strong technical support for video quality assessment.

[0156] In this embodiment of the application, an initial model is trained based on the dataset to be trained to obtain a trained video scoring model, including:

[0157] The video to be trained is input into the initial model to obtain the fused feature vector of the video to be trained;

[0158] The fused feature vector of the video to be trained is converted into one-dimensional data, and the score difference between the one-dimensional data and the actual quality score of the video to be trained is determined.

[0159] The left and right view data of the video to be trained are determined, and a first disparity loss value and a second disparity loss value are obtained based on the left and right view data of the video to be trained; wherein, the first disparity loss value represents the disparity of the left view data of the video to be trained relative to the right view data, and the second disparity loss value represents the disparity of the right view data of the video to be trained relative to the left view data.

[0160] The difference between the first disparity loss value and the second disparity loss value is determined as the loss difference.

[0161] Determine the feature difference between the fused feature vectors of adjacent frames in the video to be trained; where the feature difference represents the change between the fused feature vectors of adjacent frames.

[0162] If the score difference, loss difference, and feature difference all meet the preset difference conditions, then the initial model training is complete, and the video scoring model is obtained.

[0163] The first disparity loss value is the disparity map from the left image to the right image.

[0164] The second disparity loss value is the disparity map from the right image to the left image. Left-to-right disparity and right-to-left disparity essentially describe the same physical phenomenon: the positional difference produced when viewing the same object from different angles. For high-quality 3D video, theoretically, right-to-left disparity and left-to-right disparity are numerically equal, only in opposite directions. If the difference in loss between right-to-left and left-to-right disparity is less than a preset threshold (usually one pixel or less), then the disparities of these two pixels are considered consistent; otherwise, they are considered inconsistent.

[0165] The quality score is a score given by the user to the video to be trained.

[0166] The fused feature vector is mapped to a 1D vector, and the score difference between the 1D vector and the actual quality score of the video to be trained is obtained. The disparity consistency loss between the left and right feature views is calculated. This is achieved by calculating the difference between the disparity maps from left to right and from right to left. A simple approach is to subtract the corresponding positions of the disparity maps from left to right and from right to left to obtain the difference. This is achieved by comparing the difference in the fused feature map vectors Ff between adjacent frames, obtaining the feature difference between the fused feature vectors of adjacent frames. For example, L1 loss can be used to measure the change in feature maps between adjacent frames, thus minimizing the overall change between all adjacent frames. If the score difference, loss difference, and feature difference (small change in feature maps between all adjacent frames) are all small, the initial model training is considered complete. If the score difference, loss difference, and feature difference are arbitrarily large, the learning rate is reduced and the batch size is increased to train the model. When the differences in scores, losses, and features between the one-dimensional data and the actual quality scores of the video to be trained are all small, the initial model training is considered complete. The model combines objective image features (i.e., the evaluation between adjacent frames is represented by the loss difference, and the difference between the disparity maps from left to right and from right to left is represented by the feature difference) with subjective evaluation (i.e., the user's subjective rating is represented by the score difference) to make the model more comprehensive and accurate in evaluating video quality.

[0167] The advantage of this setup is that, through multi-dimensional feature analysis and optimization, the performance of the video scoring model is significantly improved, enabling it to evaluate video quality more accurately and comprehensively.

[0168] This application provides a video data processing method based on left and right views. By using image frames from the left eye view and image frames from the right eye view, the disparity fusion features of the video to be processed are obtained, making video evaluation more accurate and comprehensive. Simultaneously, by linking the difference between the fusion feature vector and the quality score, the disparity between the left and right view data of the training video, the disparity between the right and left view data of the training video, and the feature difference between the fusion feature vectors of adjacent frames, subjective score evaluation is added in addition to assessing the quality of the image itself, making video evaluation more comprehensive.

[0169] Figure 2 This application provides a schematic flowchart of another method for processing video data based on left and right views. Figure 2 As shown, the execution subject of this method can be a server or other servers; this embodiment does not impose any special restrictions here. Figure 2 As shown, the method includes:

[0170] S201, Receive the left view image sequence, the right view image sequence, and the text content of the stereoscopic video.

[0171] The objective quality score is a subjective evaluation of the 3D video by relevant experts.

[0172] During the training phase, the text content included information such as the type of stereoscopic video quality issue, its location, severity, and subjective rating. Specifically, it stated: "This 3D video has excellent image quality, vibrant colors, and rich detail. The 3D effect is natural, with objects exhibiting a clear sense of depth and three-dimensionality. However, some errors exist in the disparity map of certain dynamic scenes, resulting in slight imperfections in the 3D effect. Rating: 7.5 / 10." During the testing phase (i.e., the user experience phase), the text content only included the type of stereoscopic video quality issue. Specifically, it stated: "Does this 3D video cause visual discomfort or fatigue?"

[0173] During the training phase, a large amount of stereo video dataset was collected, including left and right view image sequences, corresponding text descriptions, and objective quality scores.

[0174] (2) Perform preprocessing operations such as format unification, resolution adjustment, and frame synchronization on the left and right view image sequences.

[0175] (3) Clean and label the text description, including the type, location, severity and subjective score of the quality problem.

[0176] (4) Using the trained stereo matching algorithm (RAFT-Stereo), calculate the disparity image for each corresponding left and right view image in the sequence. Post-process the disparity image, for example, by using median filtering to reduce noise.

[0177] S202. Using a stereo matching algorithm, calculate the disparity image sequence by analyzing the left view image sequence and the right view image sequence.

[0178] Among them, the stereo matching algorithm includes: RAFT-Stereo.

[0179] S203. Using a visual encoder, feature encoding is performed on the left view image sequence, the right view image sequence, and the disparity image sequence to obtain the left image feature encoding sequence, the right image feature encoding sequence, and the disparity image feature encoding sequence.

[0180] In this process, a pre-trained visual encoder is used to extract features from the left and right view image sequences and the disparity image sequence to obtain feature codes. The visual encoder can be a Vision Transformer Encoder or an encoder based on a CNN (Convolutional Neural Network) to capture the spatial context information and local detail features of the image.

[0181] S204. Align the features of the left image feature coding sequence, the right image feature coding sequence, and the disparity image feature coding sequence to ensure they are in the same coordinate system.

[0182] Specifically, a Spatial Transformation Network (STN) is used to ensure that all feature encodings are within the same coordinate system. The STN can automatically learn how to adjust the position of the feature maps to achieve better alignment.

[0183] S205. Perform disparity consistency dynamic fusion on the aligned left image feature encoding sequence, right image feature encoding sequence and disparity map feature encoding sequence to obtain the attention weight matrix.

[0184] In the fusion process, a parallax consistency constraint is introduced, and an attention mechanism is used to dynamically adjust the weights of features from different sources, so that the fused features can simultaneously reflect spatial, temporal, and depth information.

[0185] The method for dynamically fusing the aligned left-image feature encoding sequence, right-image feature encoding sequence, and disparity map feature encoding sequence to obtain a more comprehensive and richer feature representation is as follows:

[0186] (1) Aligned feature map F L (Left figure feature coding sequence), F R (Right figure feature coding sequence), and F D (Disparity map feature encoding sequence).

[0187] (2) For each corresponding position (x, y) in the feature map, calculate the disparity consistency score s(x, y) between the features of the left and right images. This can be cosine similarity or other suitable metrics. At the same time, for the disparity map FD, obtain the disparity feature value d(x, y) at that position.

[0188] The cosine similarity is:

[0189]

[0190] Where (x, y) represents any position, F L F represents the alignment vector of the left view data. R Let S(x, y) represent the alignment vector of the right view data, and let S(x, y) be the cosine similarity.

[0191] Consistency score:

[0192] s(x, y) = 1 - S(x, y)

[0193] (3) Generate an attention weight matrix based on consistency score and disparity value.

[0194] α LR (x, y)=exp(-λ·|d(x, y)|)

[0195] α RL (x, y)=exp(-μ·|d(x, y)|)

[0196] Where d(x, y) is the alignment vector of the disparity map data at position (x, y), and α LR α RL The weights are for different directions and can be pre-set constants or parameters learned by the network. λ and μ are adjustment parameters that control the weights as they change with the disparity value.

[0197] s′ LR (x, y) = s(x, y)·α LR (x, y)

[0198] Among them, s′ LR (x, y) represents the consistency score of the left view data alignment vector.

[0199] s′ RL (x, y) = s(x, y)·α RL (x, y)

[0200] Among them, s′ RL (x, y) represents the consistency score of the right view data alignment vector.

[0201] The attention weight for the left view is:

[0202]

[0203] Among them, W L (x, y) represents the attention weight of the left view, and i, j represent the view traversal positions.

[0204] The method for obtaining the attention weights of the right view is as follows:

[0205]

[0206] Among them, W R (x, y) represents the attention weights for the right view.

[0207] The method for obtaining attention weights in disparity maps is as follows:

[0208]

[0209] Among them, W D (x, y) represents the attention weights of the disparity map.

[0210] The attention weight matrix is ​​as follows:

[0211] F f (x, y) = W L (x, y) × F L (x, y) + W R (x, y) × F R (x, y) + W D (x, y) × F D (x, y)

[0212] Among them, F L (x, y) is the aligned feature encoding sequence of the left image, F R (x, y) represents the aligned right view encoded sequence size, F D (x, y) is the aligned disparity map feature encoding sequence.

[0213] During training, the attention weight matrix Ff is mapped to a 1-dimensional vector and subtracted from the objective quality score to obtain the regression loss.

[0214] Calculate the left and right feature views F L F R The disparity consistency loss between the images is calculated. This is achieved by calculating the difference between the disparity maps from left to right and from right to left, i.e., the difference between the disparity from left to right and the disparity from right to left.

[0215] Calculate the spatiotemporal consistency loss. This is achieved by comparing the Ff difference between feature maps of adjacent frames. For example, L1 loss can be used to measure the changes in feature maps between adjacent frames, resulting in a smoother overall frame variation.

[0216] Training is complete when the regression loss, disparity consistency loss, and spatiotemporal consistency loss are all relatively small.

[0217] S206. Input the attention weight matrix into the feature adaptation layer to perform visual feature mapping and obtain the mapped feature encoding.

[0218] S207. Encode the textual descriptions of video quality using textual features.

[0219] During the training phase, the text content includes information such as the type of stereoscopic video quality problem, the location of the problem, its severity, and subjective scores; during the testing phase, the text content only includes the type of stereoscopic video quality problem.

[0220] Natural language processing models (such as BERT, GPT, etc.) are used to encode the input text description, extract semantic features from the text, and generate text feature codes.

[0221] Visual feature encoding and text feature encoding are combined and input into a large language model.

[0222] The large language model is fine-tuned using labeled data to enable it to understand and integrate visual and textual information, and output accurate video quality assessments.

[0223] S208. The attention weight matrix is ​​combined with text feature encoding, and then fed into a large language model through concatenation or a more complex fusion mechanism to output a comprehensive evaluation of video quality. This large language model, after training, is able to understand and integrate visual and textual information.

[0224] During the testing phase (i.e., the user experience phase), users input a stereoscopic video and a textual description of quality issues they raise. The trained model, based on the input visual and textual information, outputs an overall evaluation of the stereoscopic video's quality, including the type of problem, its location, severity, and overall score. Specifically, the user input is: "Stereoscopic video," and the question is: "Will this 3D video cause visual discomfort or fatigue?" The answer is: "The 3D effect of this 3D video may cause visual discomfort or fatigue for some viewers, mainly due to some mismatches in the disparity map and excessively large disparity ranges. This may cause viewers to experience visual illusions, such as incorrect occlusion relationships and overly abrupt 3D effects." Score: 5.5 / 10.

[0225] This application provides another method for processing video data based on left and right views, offering a stereoscopic video quality assessment method based on a multimodal large model. By combining stereoscopic video features and a multimodal large model, it can comprehensively and meticulously evaluate the quality of stereoscopic video. Key stereoscopic features are extracted from the left and right view sequences and disparity map sequences of the stereoscopic video, and feature alignment and disparity consistency fusion are performed to construct a feature encoding that contains both detailed information and a global perspective.

[0226] In one possible implementation, it can also be used Figure 3 This represents another method for processing video data based on left and right views, such as... Figure 3As shown, left and right view sequences are obtained from the stereoscopic video. A disparity map sequence is then derived from the left and right view sequences. These sequences are then input into a visual encoder to obtain left view sequence encoding, right view sequence encoding, and disparity map sequence encoding. The left view sequence encoding, right view sequence encoding, and disparity map sequence encoding are aligned. Then, the aligned left view sequence encoding, aligned right view sequence encoding, and aligned disparity map sequence encoding undergo disparity consistency dynamic fusion processing to obtain an attention weight matrix. This attention weight matrix is ​​then input into a feature adaptation layer, along with text content. The text content is feature-encoded to obtain text feature encoding. In the feature adaptation layer, the attention weight matrix and text features are fused. Finally, the fused attention weight matrix and text features are input into a large language model to obtain a video quality assessment.

[0227] Figure 4 This is an example diagram illustrating the structure of a video data processing device based on left and right views, provided as an embodiment of this application. Figure 4 As shown, the video data processing device 40 based on left and right views includes: a first determining module 401, a second determining module 402, and a third determining module 403. Wherein:

[0228] The first determining module 401 is used to determine the left view data and right view data of the video to be processed; wherein, the left view data is the image frame of the left eye view and the right view data is the image frame of the right eye view.

[0229] The second determining module 402 is used to determine the disparity fusion features of the video to be processed based on the left view data and the right view data; wherein, the disparity fusion features represent the views of the left and right eyes and the disparity maps of the left and right eyes.

[0230] The third determining module 403 is used to determine the quality score of the video to be processed based on the parallax fusion features and the preset text description; wherein, the preset text description represents the evaluation dimension of the video to be processed, and the quality score represents the video quality of the video to be processed.

[0231] Figure 5 This is an example diagram illustrating the structural architecture of a video data processing device based on left and right views, provided as an embodiment of this application. Figure 5 As shown, the video data processing device 50 based on left and right views includes: a first determining module 501, a second determining module 502, and a third determining module 503. The second determining module 502 includes an obtaining unit 5021 and a determining unit 5022, and the third determining module 503 includes an obtaining unit 5031, wherein:

[0232] In one example, the second determining module 502 includes:

[0233] Unit 5021 is used to obtain disparity map data based on the left view data and the right view data;

[0234] The determining unit 5022 is used to determine the disparity fusion features of the video to be processed based on the left view data, the right view data, and the disparity map data.

[0235] In one possible implementation, unit 5021 is obtained, specifically used for:

[0236] The left view data, right view data, and disparity data are encoded separately to obtain the feature vectors of the left view data, right view data, and disparity data.

[0237] Based on the feature vectors of the left view data, the right view data, and the disparity map data, the disparity fusion features of the video to be processed are obtained.

[0238] In one possible implementation, unit 5021 is obtained, specifically used for:

[0239] Alignment processing is performed on the feature vectors of the left view data, the right view data, and the disparity map data to obtain the alignment vectors of the left view data, the right view data, and the disparity map data; wherein the alignment vectors of the left view data, the right view data, and the disparity map data are located in the same coordinate system;

[0240] Based on the alignment vectors of the left view data, the right view data, and the disparity map data, the disparity fusion features of the video to be processed are obtained.

[0241] In one possible implementation, unit 5021 is obtained, specifically used for:

[0242] Determine the similarity between the alignment vectors of the left view data and the alignment vectors of the right view data;

[0243] Based on similarity, the weight of the left view data is determined as the first weight, and the weight of the right view data is determined as the second weight;

[0244] The weights of the disparity map are determined based on the alignment vector of the disparity map data; these weights are designated as the third weight.

[0245] Based on the alignment vectors of the left view data, the right view data, the disparity map data, the first weight, the second weight, and the third weight, the disparity fusion features of the video to be processed are determined.

[0246] In one possible implementation, unit 5021 is obtained, specifically used for:

[0247] A first score is obtained based on the similarity and a preset first parameter; whereby the first score represents the consistency score between the left view data and the right view data.

[0248] A second score is obtained based on the similarity and a preset second parameter; whereby the second score represents the consistency score of the right view data relative to the left view data.

[0249] Based on the first score, the weight of the left view data is obtained, which is the first weight; and based on the second score, the weight of the right view data is determined, which is the second weight.

[0250] In one example, the third determining module 503 includes:

[0251] Unit 5301 is used to input the disparity fusion features and the preset text description into the preset large model to obtain the quality score of the output video to be processed.

[0252] In one possible implementation, unit 5301 is obtained, specifically used for:

[0253] Obtain the training dataset; the training dataset includes the training video, the label data of the training video, the label data representing the actual quality score of the training video, and the training text description;

[0254] The initial model is trained using the dataset to be trained to obtain a trained video scoring model. The video scoring model is used to determine the left and right view data of the video to be processed, and to determine the disparity fusion features of the video to be processed based on the left and right view data. The quality score of the video to be processed is determined based on the disparity fusion features and the preset text description.

[0255] In one possible implementation, unit 5301 is obtained, specifically used for:

[0256] The video to be trained is input into the initial model to obtain the fused feature vector of the video to be trained;

[0257] The fused feature vector of the video to be trained is converted into one-dimensional data, and the score difference between the one-dimensional data and the actual quality score of the video to be trained is determined.

[0258] The left and right view data of the video to be trained are determined, and a first disparity loss value and a second disparity loss value are obtained based on the left and right view data of the video to be trained; wherein, the first disparity loss value represents the disparity of the left view data of the video to be trained relative to the right view data, and the second disparity loss value represents the disparity of the right view data of the video to be trained relative to the left view data.

[0259] The difference between the first disparity loss value and the second disparity loss value is determined as the loss difference.

[0260] Determine the feature difference between the fused feature vectors of adjacent frames in the video to be trained; where the feature difference represents the change between the fused feature vectors of adjacent frames.

[0261] If the score difference, loss difference, and feature difference all meet the preset difference conditions, then the initial model training is complete, and the video scoring model is obtained.

[0262] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 60 includes:

[0263] The electronic device 60 may include a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a communication component 603, and other components. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0264] In the specific implementation process, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the above-described method for processing video data based on left and right views.

[0265] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0266] In the above Figure 6In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0267] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0268] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0269] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described methods for processing video data based on left and right views.

[0270] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0271] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0272] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the video data processing methods based on left and right views provided in embodiments of this application.

[0273] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0274] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0275] Since the instructions stored in the storage medium can execute the steps in any of the video data processing methods based on left and right views provided in the embodiments of this application, the beneficial effects that any of the video data processing methods based on left and right views provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0276] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0277] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0278] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0279] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0280] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0281] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0282] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0283] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0284] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0285] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0286] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method of processing video data based on left and right views, characterized by, The method comprises: determining left view data and right view data of a video to be processed; wherein the left view data is an image frame of a left eye view, and the right view data is an image frame of a right eye view; obtaining disparity map data according to the left view data and the right view data; respectively encoding the left view data, the right view data, and the disparity map data to obtain a feature vector of the left view data, a feature vector of the right view data, and a feature vector of the disparity map data; aligning the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data to obtain an aligned vector of the left view data, an aligned vector of the right view data, and an aligned vector of the disparity map data; wherein the aligned vector of the left view data, the aligned vector of the right view data, and the aligned vector of the disparity map data are located in the same coordinate system; determining a similarity between the aligned vector of the left view data and the aligned vector of the right view data; determining a weight of the left view data as a first weight and a weight of the right view data as a second weight according to the similarity; determining a weight of the disparity map as a third weight according to the aligned vector of the disparity map data; determining a disparity fusion feature of the video to be processed according to the aligned vector of the left view data, the aligned vector of the right view data, the aligned vector of the disparity map data, the first weight, the second weight, and the third weight; wherein the disparity fusion feature represents the left and right eye views and the left and right eye disparity maps; determining a quality score of the video to be processed according to the disparity fusion feature and a preset text description; wherein the preset text description represents an evaluation dimension of the video to be processed, and the quality score represents a video quality of the video to be processed.

2. The method of claim 1, wherein, The determining of the weight of the left view data as a first weight and the weight of the right view data as a second weight according to the similarity comprises: obtaining a first score according to the similarity and a preset first parameter; wherein the first score represents a consistency score of the left view data relative to the right view data; obtaining a second score according to the similarity and a preset second parameter; wherein the second score represents a consistency score of the right view data relative to the left view data; determining the weight of the left view data as a first weight according to the first score, and determining the weight of the right view data as a second weight according to the second score.

3. The method of claim 1, wherein, The determining of the quality score of the video to be processed according to the disparity fusion feature and the preset text description comprises: inputting the disparity fusion feature and the preset text description into a preset large model to obtain an output quality score of the video to be processed.

4. The method according to any one of claims 1 to 3, characterized in that, Further comprising: obtaining a training data set; wherein the training data set comprises a training video, label data of the training video, the label data representing an actual quality score of the training video and a training text description. The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description.

5. The method of claim 4, wherein, The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description. The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description. The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description. The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description. The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description. The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description. The initial model is trained according to the to-be-trained data set, and a trained video scoring model is obtained; wherein the video scoring model is used to determine left view data and right view data of a to-be-processed video, determine a parallax fusion feature of the to-be-processed video according to the left view data and the right view data, and determine a quality score of the to-be-processed video according to the parallax fusion feature and a preset text description.

6. A processing apparatus of video data based on left and right views, characterized by, ​ ​ The second determining module is configured to obtain disparity map data according to the left view data and the right view data; encode the left view data, the right view data, and the disparity map data respectively to obtain a feature vector of the left view data, a feature vector of the right view data, and a feature vector of the disparity map data; perform alignment processing on the feature vector of the left view data, the feature vector of the right view data, and the feature vector of the disparity map data to obtain an aligned vector of the left view data, an aligned vector of the right view data, and an aligned vector of the disparity map data; the aligned vector of the left view data, the aligned vector of the right view data, and the aligned vector of the disparity map data are located in the same coordinate system; determine a similarity between the aligned vector of the left view data and the aligned vector of the right view data; determine the weight of the left view data as a first weight and determine the weight of the right view data as a second weight according to the similarity; determine a weight of the disparity map as a third weight according to the aligned vector of the disparity map data; determine a disparity fusion feature of the video to be processed according to the aligned vector of the left view data, the aligned vector of the right view data, the aligned vector of the disparity map data, the first weight, the second weight, and the third weight; the disparity fusion feature represents the left and right eye views and the left and right eye disparity maps. The third determining module is configured to determine a quality score of the video to be processed according to the disparity fusion feature and a preset text description; the preset text description represents an evaluation dimension of the video to be processed, and the quality score represents a video quality of the video to be processed.

7. An electronic device, comprising: The method comprises: a processor and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1 to 5.

9. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 5.

Images