Video quality analysis method and device, equipment, storage medium and program product

Abstract

The invention provides a video quality analysis method and device, equipment, a storage medium and a program product, and relates to the technical field of computers, and the method comprises the steps: extracting target data from video data to be subjected to quality analysis; wherein the target data comprises video frame data, text data and audio data; performing feature extraction processing on the target data to obtain a plurality of target features corresponding to the target data; performing feature fusion processing on the plurality of target features to obtain a multi-dimensional feature fusion result; and determining a video quality analysis result corresponding to the video data according to the multi-dimensional feature fusion result. The accuracy of a video quality analysis result can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to video quality analysis methods, apparatus, devices, storage media, and program products. Background Technology

[0002] In related technologies, when reviewing video content, it is often necessary to use a visual language model to output the video quality analysis results of the video content through a binary classification method based on the video content.

[0003] However, binary classification is easily affected by thresholds. In other words, thresholds directly affect the binary classification judgment. If the threshold is set unreasonably, it may lead to inaccurate video quality analysis results. Summary of the Invention

[0004] This disclosure provides a video quality analysis method, apparatus, device, storage medium, and program product to solve the problem of inaccurate review results caused by the binary classification method for reviewing video content in related technologies.

[0005] In a first aspect, this disclosure provides a video quality analysis method, which includes: extracting target data from video data to be analyzed; wherein the target data includes video frame data, text data, and audio data; performing feature extraction processing on the target data to obtain multiple target features corresponding to the target data; performing feature fusion processing on the multiple target features to obtain a multi-dimensional feature fusion result; and determining the video quality analysis result corresponding to the video data based on the multi-dimensional feature fusion result.

[0006] In one optional implementation, feature fusion processing is performed on multiple target features to obtain a multi-dimensional feature fusion result, including: At least two target features with the same target type are merged into a first target feature group; wherein, the target type includes video frame type, text type and audio type; Multiple first target feature groups are fused to obtain multi-dimensional feature fusion results.

[0007] In one optional implementation, multiple first target feature groups are fused to obtain a multi-dimensional feature fusion result, including: Predict a second target feature group that corresponds one-to-one with the first target feature group; wherein the second target feature group is obtained by predicting at least two first target feature groups; Based on the first target feature group and the second target feature group, the multi-dimensional feature fusion result is determined.

[0008] In one optional implementation, the first target feature group is represented as a first feature capsule, and the second target feature group is represented as a second feature capsule; and the multi-dimensional feature fusion result is determined based on the first target feature group and the second target feature group, including: Based on the pose vector of the first feature capsule, the pose vector of the first feature capsule is mapped to the target vector of the second feature capsule; Based on the first feature capsule, a first coupling coefficient for the second feature capsule is determined; wherein the first coupling coefficient indicates the contribution of the first feature capsule to the second feature capsule; The multi-dimensional feature fusion result is determined based on the first coupling coefficient and multiple first feature capsules.

[0009] In one optional implementation, the multi-dimensional feature fusion result is determined based on a first coupling coefficient and multiple first feature capsules, including: Based on the first coupling coefficient and multiple first feature capsules, the output value of the second target feature group is determined; wherein, the output value indicates the probability of the existence of the second feature capsule; Based on the output value of the second feature capsule, update the first coupling coefficient to obtain the second coupling coefficient; Based on the second coupling coefficient, the steps of determining the output value of the second target feature group based on the first coupling coefficient and multiple first feature capsules, updating the first coupling coefficient based on the output value of the second feature capsule, and obtaining the second coupling coefficient are repeated until the number of repetitions reaches the threshold, and the multi-dimensional feature fusion result is determined.

[0010] In one alternative implementation, the first feature capsule includes: a visual semantic capsule, a visual motion capsule, a textual semantic capsule, a paralinguistic feature capsule, and a title capsule.

[0011] In one optional implementation, feature extraction processing is performed on the target data to obtain multiple target features corresponding to the target data, including: Using a pre-trained image classification task model, keyframes and transition frames are determined based on video frame data; where video frame data is the input of the pre-trained image classification task model, and keyframes and transition frames are the outputs of the pre-trained image classification task model. Using a pre-trained semantic feature extraction model, the visual semantic features corresponding to the key frames are determined based on the key frames; where the key frames are the input of the pre-trained semantic feature extraction model, and the visual semantic features are the output of the pre-trained semantic feature extraction model. Using a pre-trained convolutional neural network model, the visual motion features corresponding to the transition frames are determined based on the transition frames; where the transition frames are the input of the pre-trained convolutional neural network model, and the visual motion features are the output of the pre-trained convolutional neural network model.

[0012] In one optional implementation, feature extraction processing is performed on the target data to obtain multiple target features corresponding to the target data, including: Using a pre-trained language recognition model, word-level embeddings and sentence-level embeddings are determined based on the text data. The text data is the input of the pre-trained language recognition model, and the word-level and sentence-level embeddings are the outputs of the pre-trained language recognition model. The word-level and sentence-level embeddings indicate the features corresponding to the text data.

[0013] In one optional implementation, feature extraction processing is performed on the target data to obtain multiple target features corresponding to the target data, including: Local feature extraction is performed on the audio data to determine the corresponding non-textual data; the non-textual data includes at least: intonation data, speech rate data, and emotional tone data; Using a pre-trained long short-term memory network model, the temporal evolution pattern information corresponding to non-textual data is determined based on the non-textual data; where the non-textual data is the input of the pre-trained long short-term memory network model, and the temporal evolution pattern information is the output of the pre-trained long short-term memory network model. Based on the temporal evolution pattern information, extract the corresponding paralinguistic features.

[0014] In one optional implementation, the video quality analysis result corresponding to the video data is determined based on the multi-dimensional feature fusion result, including: Using a pre-trained recognition model, the video quality analysis result corresponding to the video data is determined based on the multi-dimensional feature fusion result; where the multi-dimensional feature fusion result is the input of the pre-trained recognition model, and the video quality analysis result is the output of the pre-trained recognition model.

[0015] In one optional implementation, the video quality analysis method further includes: The video data is input into a pre-trained recognition model to obtain the predicted review result corresponding to the video data; where the video data is the input of the pre-trained recognition model and the predicted review result is the output of the pre-trained recognition model. Based on the predicted review results and the labeled review results, the mean squared error and text similarity are determined; The parameters of the pre-trained recognition model are updated based on the mean squared error and text similarity.

[0016] In one optional implementation, the video quality analysis method further includes: Obtain the first training data; wherein, the first training data includes: the video data to be trained, and the first multi-dimensional feature fusion result to be trained corresponding to the video data to be trained; Based on the video data to be trained and the fusion result of the first multi-dimensional features to be trained, the neural network model is trained to obtain a pre-trained recognition model. The first training data is generalized to obtain the second training data and the fusion result of the second multi-dimensional features to be trained; wherein, the number of second training data is greater than the number of first training data. The pre-trained recognition model is retrained using the second training data and the fusion result of the second multi-dimensional features to be trained.

[0017] In one optional implementation, the video quality analysis method further includes: Obtain the feedback results corresponding to the video data; When the feedback result indicates that the video data does not meet the preset requirements, the pre-trained recognition model is retrained based on the feedback result corresponding to the video data.

[0018] Secondly, this disclosure provides a video quality analysis apparatus, comprising: an extraction module for extracting target data from video data to be analyzed; wherein the target data includes video frame data, text data, and audio data; a feature extraction processing module for performing feature extraction processing on the target data to obtain multiple target features corresponding to the target data; a feature fusion processing module for performing feature fusion processing on the multiple target features to obtain a multi-dimensional feature fusion result; and a determination module for determining the video quality analysis result corresponding to the video data based on the multi-dimensional feature fusion result.

[0019] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the video quality analysis method described in the first aspect or any corresponding embodiment.

[0020] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the video quality analysis method described in the first aspect or any corresponding embodiment thereof.

[0021] Fifthly, this disclosure provides a computer program product, including computer instructions for causing a computer to execute the video quality analysis method described in the first aspect or any corresponding embodiment thereof.

[0022] The video quality analysis method provided in this disclosure takes into account the following situation: the video quality analysis results output by binary classification are easily affected by the threshold. If the threshold setting is unreasonable, it may lead to inaccurate video quality analysis results.

[0023] This disclosure takes into account the above-mentioned situations, extracting video frame data, text data, and audio data from the video data to be analyzed for quality. Feature extraction processing is then performed on the extracted target data to obtain multiple target features, followed by feature fusion processing to obtain a multi-dimensional feature fusion result. Because multi-dimensional feature fusion comprehensively considers the multi-dimensional feature information of the video data to be analyzed, rather than simply performing binary classification, it avoids the one-sidedness of classifying based solely on a single feature and a set threshold. Instead, it determines the video quality analysis result by comprehensively analyzing the multi-dimensional feature fusion result, thereby improving the accuracy of the video quality analysis result. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure; Figure 2 This is the first flowchart of the video quality analysis method provided according to the embodiments of this disclosure; Figure 3 This is a second flowchart of the video quality analysis method provided according to embodiments of this disclosure; Figure 4 This is the third flowchart of the video quality analysis method provided according to the embodiments of this disclosure; Figure 5 This is the fourth flowchart of the video quality analysis method provided according to the embodiments of this disclosure; Figure 6 This is the fifth flowchart of the video quality analysis method provided according to the embodiments of this disclosure; Figure 7 This is a structural block diagram of a video quality analysis apparatus according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise expressly specified.

[0029] As one optional application scenario of this disclosure embodiment, such as Figure 1 As shown, the video quality analysis system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0030] Specifically, the terminal device can be a smartphone, tablet, laptop, PDA, desktop computer, game console, smart TV, smart wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.

[0031] Taking video quality analysis as an example, the terminal device has a video quality analysis application installed. By running this video quality analysis application, the video data to be analyzed can be performed to obtain the video quality analysis results.

[0032] In some technologies, video quality analysis results are often output through binary classification, which is easily affected by thresholds. If the threshold is set unreasonably, it may lead to inaccurate video quality analysis results.

[0033] The video quality analysis method provided in this disclosure extracts video frame data, text data, and audio data from the video data to be analyzed. It then performs feature extraction processing on the extracted target data to obtain multiple target features, and finally performs feature fusion processing to obtain a multi-dimensional feature fusion result. Because multi-dimensional feature fusion comprehensively considers the multi-dimensional feature information of the video data to be analyzed, rather than simply performing binary classification, it avoids the one-sidedness of classifying based solely on a single feature and a set threshold. Instead, it determines the video quality analysis result by comprehensively analyzing the multi-dimensional feature fusion result, thereby improving the accuracy of the video quality analysis result.

[0034] According to an embodiment of the present invention, a video quality analysis method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This embodiment provides a video quality analysis method, which can be used in the aforementioned terminal devices, such as computer 101, mobile terminal 102, etc. Figure 2 This is the first flowchart of the video quality analysis method provided according to the embodiments of this disclosure, such as... Figure 2 As shown, the process includes the following steps: Step S201: Extract target data from the video data to be analyzed for quality; wherein, the target data includes video frame data, text data and audio data.

[0036] The video data to be analyzed can be video data uploaded by the user to the terminal device, or video data stored in the terminal device, etc., without specific limitations.

[0037] As an example, users can upload video data for quality analysis by typing on the keyboard, clicking the mouse, or touching the screen. The terminal device can respond to the upload operation to obtain the video data for quality analysis.

[0038] Target data can include video frame data, text data, and audio data. For example, when extracting video frame data from video data to be analyzed for quality, the video frame data can be the target data. Similarly, when extracting text data from video data to be analyzed for quality, the text data can be the target data.

[0039] After obtaining the video data to be analyzed, the target data can be extracted from the video data.

[0040] As an example, for video frame data, relevant techniques (such as histogram-based difference method, optical flow method, etc.) can be used to extract video frame data from the video data to be analyzed for quality.

[0041] As an example, a pre-trained deep learning model can be used to determine video frame data based on the video data to be analyzed for quality. The video data to be analyzed can be the input to the pre-trained deep learning model, and the video frame data can be the output of the pre-trained deep learning model.

[0042] As an example, for text data, optical character recognition methods can be used to extract text data from video data to be analyzed for quality.

[0043] As an example, for audio data, the text can be output by an automatic language recognition model, and then the semantics can be analyzed using natural language processing techniques to obtain the audio data.

[0044] Step S202: Perform feature extraction processing on the target data to obtain multiple target features corresponding to the target data.

[0045] As discussed above, target data can include video frame data, text data, and audio data. Correspondingly, the multiple target features corresponding to the target data can indicate the target features corresponding to the video frame data, the target features corresponding to the text data, and the target features corresponding to the audio data.

[0046] As an example, a pre-trained neural network model can be used to obtain multiple target features corresponding to the target data. Here, the target data serves as the input to the pre-trained neural network model, and the multiple target features corresponding to the target data can be the output of the pre-trained neural network model. Other methods can also be used to determine the multiple target features corresponding to the target data; no specific limitations are imposed here, and these can be implemented by those skilled in the art.

[0047] Step S203: Perform feature fusion processing on multiple target features to obtain multi-dimensional feature fusion results.

[0048] Multi-dimensional feature fusion results can indicate the outcome of fusing multiple target features. Among them, multi-dimensional feature fusion results have higher-dimensional semantic information and can capture multiple aspects of target features simultaneously.

[0049] After identifying multiple target features, feature fusion processing can be performed on these multiple target features to obtain multi-dimensional feature fusion results.

[0050] As an example, a multi-dimensional feature fusion result can be obtained based on a neural network model and multiple target features.

[0051] As an example, multiple target features can be fused using contrastive learning or alignment loss to obtain multi-dimensional feature fusion results.

[0052] Step S204: Based on the multi-dimensional feature fusion results, determine the video quality analysis results corresponding to the video data.

[0053] Video quality analysis results can indicate whether the video data is qualified, and the reasons why the video data is unqualified.

[0054] For example, the video quality analysis results can indicate the following: the video data to be analyzed has high information density, high knowledge value, and a rigorous logical structure; the video data to be analyzed has a resolution >720p, natural transitions, accurate subtitles, and stable transitions; the video data to be analyzed lacks depth in the corresponding field; the video data to be analyzed is plagiarized and lacks innovative presentation; the final video quality analysis score is 60 points.

[0055] After determining the multi-dimensional feature fusion results, the corresponding video quality analysis results of the video data can be determined based on the multi-dimensional feature fusion results.

[0056] As an example, a correlation table between the multi-dimensional feature fusion results and the video quality analysis results can be pre-defined. When the multi-dimensional feature fusion results are determined, the video quality analysis results can be determined by querying the correlation table.

[0057] As an example, a quality analysis result model can be used to determine the video quality analysis result corresponding to the video data based on the multi-dimensional feature fusion result. Here, the multi-dimensional feature fusion result is the input to the quality analysis result model, and the video quality analysis result is the output of the quality analysis result model.

[0058] The video quality analysis method provided in this disclosure extracts video frame data, text data, and audio data from the video data to be analyzed. It then performs feature extraction processing on the extracted target data to obtain multiple target features, and finally performs feature fusion processing to obtain a multi-dimensional feature fusion result. Because multi-dimensional feature fusion comprehensively considers the multi-dimensional feature information of the video data to be analyzed, rather than simply performing binary classification, it avoids the one-sidedness of classifying based solely on a single feature and a set threshold. Instead, it determines the video quality analysis result by comprehensively analyzing the multi-dimensional feature fusion result, thereby improving the accuracy of the video quality analysis result.

[0059] This embodiment provides a video quality analysis method, which can be used in the aforementioned terminal devices, such as computer 101, mobile terminal 102, etc. Figure 3 This is a second flowchart of the video quality analysis method provided according to embodiments of this disclosure, such as... Figure 3 As shown, the process includes the following steps: Step S301: Extract target data from the video data to be analyzed for quality; the target data includes video frame data, text data, and audio data. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0060] Step S302 involves feature extraction processing of the target data to obtain multiple target features corresponding to the target data. For details, please refer to [link to relevant documentation]. Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0061] Step S303: Perform feature fusion processing on multiple target features to obtain multi-dimensional feature fusion results.

[0062] Specifically, step S303 includes: Step S3031: Merge at least two target features with the same target type into a first target feature group; wherein, the target type includes video frame type, text type and audio type.

[0063] The target type indicates the type of target features. Target types include video frame type, text type, and audio type.

[0064] The first target feature group can refer to a set composed of at least two target features of the same target type. For example, the ResNet features of three consecutive frames of video data can be concatenated to form the first target feature group; another example is the average of the BERT features of five sentences of text data to form the first target feature group.

[0065] After identifying multiple target features, these features can be classified to obtain target features of video frame type, target features of text type, and target features of audio type. Then, at least two target features with the same target type are merged into the first target feature group.

[0066] As an example, a classification model can be used to determine a first target feature group based on multiple target features. The multiple target features can be the input to the classification model, and the first target feature group can be the output of the classification model.

[0067] Step S3032: Fuse multiple first target feature groups to obtain multi-dimensional feature fusion results.

[0068] After identifying multiple first target feature groups, feature fusion processing can be performed on these multiple first target feature groups to obtain multi-dimensional feature fusion results.

[0069] As an example, a multi-dimensional feature fusion result can be obtained based on a neural network model and multiple first target feature groups.

[0070] As an example, multiple first-target feature groups can be fused using contrastive learning or alignment loss to obtain a multi-dimensional feature fusion result.

[0071] Specifically, step S3032 includes: Step a1: Predict a second target feature group that corresponds one-to-one with the first target feature group; wherein the second target feature group is obtained by predicting at least two first target feature groups.

[0072] The second target feature group is predicted from at least two first target feature groups. As mentioned above, the first target feature group corresponds to three types: video frame type, text type, and audio type. Therefore, the first target feature group can also be of at least three types. When predicting the second target feature group, it needs to be obtained by predicting from at least two first target feature groups. For example, the second target feature group corresponding to the first target feature group of video frame type can be predicted using the first target feature group of video frame type and the first target feature group of text type.

[0073] Step a2: Determine the multi-dimensional feature fusion result based on the first target feature group and the second target feature group.

[0074] After determining the first target feature group and the second target feature group, the multi-dimensional feature fusion result can be further determined.

[0075] As an example, a neural network model can be used to determine the multi-dimensional feature fusion result based on the first target feature group and the second target feature group. Other methods can also be used to determine the multi-dimensional feature fusion result based on the first target feature group and the second target feature group. No specific limitation is made here, and it can be implemented by those skilled in the art.

[0076] Specifically, the first target feature group is represented as a first feature capsule, and the second target feature group is represented as a second feature capsule; and step a2 above includes: Step a21: Based on the pose vector of the first feature capsule, map the pose vector of the first feature capsule to the target vector of the second feature capsule.

[0077] The first feature capsule can be indicated as a structured feature unit composed of at least two original features of the same target type, containing a pose vector. The pose vector can indicate the geometric attributes (such as position, orientation, scale) or semantic attributes (such as emotion intensity, action category) of the first feature capsule.

[0078] The second feature capsule can be designated as a higher-level feature capsule within the target features, generated from the first feature capsule through a dynamic routing mechanism. The second feature capsule can be used to represent more abstract semantic or global features (e.g., a "kicking motion" in video data rather than a single frame's "leg movement").

[0079] The target vector can indicate the desired pose vector of the second feature capsule, which is obtained by linear transformation mapping from the pose vector of the first feature capsule.

[0080] As an example, a learnable transformation matrix can be pre-constructed, and then the target vector can be determined based on the product of the transformation matrix and the attitude vector.

[0081] Step a22: Determine the first coupling coefficient of the second feature capsule based on the first feature capsule; wherein the first coupling coefficient indicates the contribution of the first feature capsule to the second feature capsule.

[0082] The first coupling coefficient represents the contribution weight of the first feature capsule to the second feature capsule, and is updated iteratively through a dynamic routing algorithm. After determining the first feature capsule, the first coupling coefficient of the second feature capsule can be determined based on the first feature capsule.

[0083] As an example, the softmax function is used to determine the first coupling coefficient of the second feature capsule based on the first feature capsule.

[0084] Step a23: Determine the multi-dimensional feature fusion result based on the first coupling coefficient and multiple first feature capsules.

[0085] After determining the first coupling coefficient and multiple first feature capsules, the multi-dimensional feature fusion result can be determined by updating the first coupling coefficient.

[0086] Specifically, step a23 above includes: Step a231: Determine the output value of the second target feature group based on the first coupling coefficient and multiple first feature capsules; wherein the output value indicates the probability of the existence of the second feature capsule.

[0087] The output value can be understood as the activation value of the second feature capsule. Specifically, the output value represents the probability of the feature capsule existing.

[0088] Specifically, the output value of the second target feature group is determined based on the first coupling coefficient and multiple first feature capsules.

[0089] As an example, the first coupling coefficient and multiple first feature capsules can be scaled and normalized to determine the output value of the second target feature group.

[0090] Step a232: Update the first coupling coefficient based on the output value of the second feature capsule to obtain the second coupling coefficient.

[0091] The second coupling coefficient can indicate the updated coupling coefficient of the first coupling coefficient. After determining the output value of the second feature capsule, the first coupling coefficient can be updated based on the output value of the second feature capsule to obtain the second coupling coefficient.

[0092] Step a233: Based on the second coupling coefficient, repeatedly execute the steps of determining the output value of the second target feature group based on the first coupling coefficient and multiple first feature capsules, updating the first coupling coefficient based on the output value of the second feature capsule, and obtaining the second coupling coefficient, until the number of repetitions reaches the threshold, and determine the multi-dimensional feature fusion result.

[0093] Calculate the output of the second feature capsule based on the current coupling coefficient. Update the coupling coefficient based on the output. After each iteration, check if the threshold number of iterations has been reached.

[0094] If the threshold number of repetitions is reached, then the repeated execution process ends, and the final output multi-dimensional feature fusion result can be obtained.

[0095] Step S304: Based on the multi-dimensional feature fusion results, determine the video quality analysis results corresponding to the video data. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0096] The video quality analysis method provided in this disclosure, before determining the multi-dimensional feature fusion result, merges at least two target features with the same target type into a first target feature group. On the one hand, this solves the problem that target features of the same type may have duplicate information, which leads to a decrease in the utilization rate of target features. On the other hand, by merging target features with the same target type into a first target feature group, the accuracy of video quality analysis results can be improved.

[0097] Furthermore, since the pose vector itself contains various attribute information of the features, such as direction, scale, and position, by mapping the pose vector of the first feature capsule to the target vector of the second feature capsule, the information exchange between different feature capsules can be made smoother, adapting to the needs of different feature combinations and fusions, thereby improving the accuracy of video quality analysis results.

[0098] In one possible implementation, the first feature capsule includes: a visual semantic capsule, a visual motion capsule, a textual semantic capsule, a paralinguistic feature capsule, and a title capsule.

[0099] Please refer to Figure 4 , Figure 4 This is a third flowchart illustrating the video quality analysis method provided according to embodiments of this disclosure.

[0100] Combination Figure 4 As shown, the visual semantic capsule, visual motion capsule, text semantic capsule, paralinguistic feature capsule, and title capsule can be low-level capsules (i.e., the first feature capsule). After obtaining the visual semantic capsule, visual motion capsule, text semantic capsule, paralinguistic feature capsule, and title capsule, the weights of the visual semantic capsule, visual motion capsule, text semantic capsule, paralinguistic feature capsule, and title capsule can be initialized. Then, the target vector of the high-level capsule is calculated, and the coupling coefficient of the low-level capsule is calculated. Based on the coupling coefficient, all target vectors are weighted and summed to obtain the initial state of the high-level capsule. The initial state is compressed and output to determine the output value. The coupling coefficient is updated based on the output value. The iteration is repeated until the number of iterations reaches the threshold, and then the multi-dimensional feature fusion result is determined.

[0101] This embodiment provides a video quality analysis method, which can be used in the aforementioned terminal devices, such as computer 101, mobile terminal 102, etc. Figure 5 This is the fourth flowchart of the video quality analysis method provided according to the embodiments of this disclosure, such as... Figure 5 As shown, the process includes the following steps: Step S501: Extract target data from the video data to be analyzed for quality; the target data includes video frame data, text data, and audio data. For details, please refer to [link to relevant documentation]. Figure 3Step S301 of the illustrated embodiment will not be described again here.

[0102] Step S502: Perform feature extraction processing on the target data to obtain multiple target features corresponding to the target data.

[0103] Step S5021: Using a pre-trained image classification task model, determine key frames and transition frames based on video frame data; wherein, video frame data is the input of the pre-trained image classification task model, and key frames and transition frames are the outputs of the pre-trained image classification task model.

[0104] A pre-trained image classification task model can instruct a lightweight pre-analysis network. After determining the pre-trained image classification task model and video frame data, the video frame data can be input into the pre-trained image classification task model to obtain keyframes and transition frames. Keyframes can be frames that define key actions, states, or scenes in an animation or video. Transition frames can be frames that fill the gaps between keyframes to smoothly transition between key actions.

[0105] Step S5022: Using a pre-trained semantic feature extraction model, determine the visual semantic features corresponding to the key frames based on the key frames; wherein, the key frames are the input of the pre-trained semantic feature extraction model, and the visual semantic features are the output of the pre-trained semantic feature extraction model.

[0106] Visual semantic features are a core concept in computer vision and deep learning, referring to abstract features extracted from images or videos that can express high-level semantic information. After determining keyframes, the keyframes can be input into a pre-trained semantic feature extraction model, and then the keyframes can be processed according to the pre-trained semantic feature extraction model to obtain the visual semantic features corresponding to the keyframes.

[0107] Step S5023: Using a pre-trained convolutional neural network model, determine the visual motion features corresponding to the transition frame based on the transition frame; wherein, the transition frame is the input of the pre-trained convolutional neural network model, and the visual motion features are the output of the pre-trained convolutional neural network model.

[0108] Visual motion features are a core concept in computer vision and video analytics used to describe the dynamic information of objects or scenes changing over time. By capturing the motion patterns of pixels, objects, or scenes in a time series, they extend static visual information to the spatiotemporal dimension, thereby supporting the understanding and analysis of dynamic behavior. After determining the transition frame, it can be input into a pre-trained convolutional neural network model, and processed according to the model to obtain the corresponding visual motion features.

[0109] As an example, the specific implementation process can be represented as follows: K, T = PreAnalysisNet(V); where PreAnalysisNet is the pre-analysis network, V is the input data (such as video frame data), and K and T are the two output branches. K represents keyframes, and T represents transition frames.

[0110] F_semantic = ViT_Encoder(K); where ViT_Encoder is the Vision Transformer encoder and F_semantic is the visual semantic feature.

[0111] F_motion = LightCNN_Encoder(T); where LightCNN_Encoder is a lightweight CNN encoder and F_motion is the visual motion feature.

[0112] Step S503 involves performing feature fusion processing on multiple target features to obtain a multi-dimensional feature fusion result. For details, please refer to [link to relevant documentation]. Figure 3 Step S303 of the illustrated embodiment will not be described again here.

[0113] Step S504: Based on the multi-dimensional feature fusion results, determine the video quality analysis results corresponding to the video data. For details, please refer to [link to relevant documentation]. Figure 3 Step S304 of the illustrated embodiment will not be described again here.

[0114] The video quality analysis method provided in this disclosure uses a pre-trained image classification task model, semantic feature extraction model, and convolutional neural network model to process video frame data. The image classification task model quickly filters out keyframes and transition frames, while the semantic feature extraction model and convolutional neural network model extract corresponding features for keyframes and transition frames respectively. The models do not interfere with each other, shortening the overall processing time and improving the efficiency of video analysis.

[0115] In one possible implementation, step S502 above includes: Step b1: Using a pre-trained language recognition model, determine the word-level embedding and sentence-level embedding corresponding to the text data based on the text data; wherein, the text data is the input of the pre-trained language recognition model, and the word-level embedding and sentence-level embedding are the outputs of the pre-trained language recognition model, and the word-level embedding and sentence-level embedding indicate the features corresponding to the text data.

[0116] Word-level embeddings can instruct each word or subword in a text to be mapped to a low-dimensional dense vector (e.g., 300-dimensional), capturing the semantic and syntactic features of words. Sentence-level embeddings can instruct the encoding of an entire sentence or paragraph into a fixed-length vector, capturing the global semantic and structural information of the sentence.

[0117] Specifically, after determining the text data, the text data can be input into a pre-trained language recognition model, and then the text data can be processed to obtain the word-level embedding and sentence-level embedding corresponding to the text data.

[0118] As an example, word-level embeddings and sentence-level embeddings can be determined as follows: F_text=BERT(T_ASR); where BERT is a pre-trained language recognition model, BidirectionalEncoder Representations from Transformers, T_ASR is the text data, and F_text is used to indicate word-level embeddings and sentence-level embeddings.

[0119] F_title = BERT(T_title); where T_title is the title data. F_title is the title feature.

[0120] The video quality analysis method provided in this disclosure can accurately reflect the actual meaning of words by assigning different vector representations to words based on their usage in different contexts through word-level embedding. Sentence-level embedding can understand the semantics of various complex sentence structures, such as conditional sentences, interrogative sentences, and compound sentences, providing accurate feature input for subsequent natural language processing tasks, thereby accurately determining the video quality analysis results.

[0121] In one possible implementation, step S502 above includes: Step c1 involves extracting local features from the audio data to determine the non-textual data corresponding to the audio data; the non-textual data includes at least: intonation data, speech rate data, and emotional color data.

[0122] After determining the audio data, local feature extraction can be performed to identify the corresponding non-textual data. Non-textual data includes at least: intonation data, speech rate data, and emotional tone data.

[0123] As an example, the audio is framed and then subjected to Fourier transform to generate a spectrogram (speech spectrogram), which captures frequency changes over time to identify non-textual data.

[0124] As an example, the characteristics of human hearing are simulated using a Mel-scale filter bank to extract features such as fundamental frequency and harmonic structure in order to identify non-textual data.

[0125] Step c2: Using a pre-trained long short-term memory network model, determine the temporal evolution pattern information corresponding to the non-textual data based on the non-textual data; wherein, the non-textual data is the input of the pre-trained long short-term memory network model, and the temporal evolution pattern information is the output of the pre-trained long short-term memory network model.

[0126] The information on the temporal evolution pattern can be understood as: the rise and fall pattern of tone over time, and the acceleration and deceleration trend of speech rate.

[0127] After identifying the non-textual data, it can be input into a pre-trained long short-term memory network model. The pre-trained long short-term memory network model can process the non-textual data to obtain information on the temporal evolution pattern.

[0128] Step c3: Extract the sub-linguistic features corresponding to the temporal evolution pattern information based on the temporal evolution pattern information.

[0129] Paralinguistic features can be understood as means of expression other than spoken language, including spoken language (intonation, stress, pauses) and non-verbal language (facial expressions, gestures, body language).

[0130] After determining the temporal evolution pattern information, the corresponding paralinguistic features can be directly identified from the temporal evolution pattern information.

[0131] Please refer to Figure 6 , Figure 6 This is a fifth flowchart illustrating the video quality analysis method provided according to embodiments of this disclosure.

[0132] Combination Figure 6As shown, the target data includes video frame data, text data, and audio data. The target data can be input, and then processed separately for each of the video frame data, text data, and audio data. For video frame data, a lightweight pre-analysis network (i.e., the pre-trained image classification task model mentioned above) can be used to obtain keyframes and transition frames. The keyframes are then processed by a VIT encoder (i.e., the pre-trained semantic feature extraction model mentioned above) to obtain visual semantic features. Transition frames are processed by a lightweight CNN encoder (i.e., the pre-trained convolutional neural network model mentioned above) to obtain visual motion features. For text data, a BERT encoder (i.e., the pre-trained language recognition model mentioned above) is used to process the text data to obtain word-level embeddings and sentence-level embeddings (i.e., text semantic features). For audio data, Mel spectrogram transformation is used to extract local features from the audio data to obtain non-text data. Then, a 1D-CNN+LSTM (i.e., the pre-trained long short-term memory network model mentioned above) is used to determine the temporal evolution pattern information. Based on the temporal evolution pattern information, the corresponding paralinguistic features are extracted.

[0133] The video quality analysis method provided in this disclosure extracts local features from audio data, enabling the extraction of important information beyond textual content from multiple perspectives. Intonation data reflects the speaker's pitch variations; different intonations may convey different semantics and emotions such as questioning, affirmation, and surprise. Speech rate data reflects the pace of speech; a fast speech rate may indicate emotional excitement or time pressure, while a slow speech rate may suggest contemplation or a serious attitude. Emotional coloring data is directly related to the speaker's emotional state, such as joy, anger, or sadness. By simultaneously extracting these non-textual data, a more comprehensive understanding of the information contained in the audio can be achieved, avoiding information gaps caused by relying solely on textual content.

[0134] In one possible implementation, step S504 above includes: Step d1: Using a pre-trained recognition model, determine the video quality analysis result corresponding to the video data based on the multi-dimensional feature fusion result; wherein, the multi-dimensional feature fusion result is the input of the pre-trained recognition model, and the video quality analysis result is the output of the pre-trained recognition model.

[0135] After determining the multi-dimensional feature fusion result, the multi-dimensional feature fusion result can be input into a pre-trained recognition model. The pre-trained recognition model can process the multi-dimensional feature fusion result to obtain the video quality analysis result corresponding to the video data.

[0136] The video quality analysis method provided in this disclosure incorporates multi-dimensional feature fusion, covering multiple key aspects of video, such as image clarity, color reproduction, frame rate stability, audio quality, and content coherence. The pre-trained recognition model can comprehensively consider these different dimensions of feature information, avoiding the bias and errors that may arise from single-feature analysis. For example, judging video quality solely based on image clarity may ignore the impact of audio noise on the overall experience, while multi-dimensional feature fusion provides a comprehensive evaluation, resulting in more accurate video quality analysis results.

[0137] In one possible implementation, the above method also includes: Step e1: Input the video data into the pre-trained recognition model to obtain the prediction review result corresponding to the video data; wherein, the video data is the input of the pre-trained recognition model, and the prediction review result is the output of the pre-trained recognition model.

[0138] After determining the video data, it can be input into a pre-trained recognition model. The pre-trained recognition model can process the video data and obtain a predicted review result.

[0139] Step e2: Based on the predicted review results and the labeled review results, determine the mean squared error and text similarity.

[0140] After determining the predicted review results, the mean squared error and text similarity can be further determined based on the predicted review results and the labeled review results.

[0141] Step e3: Update the parameters of the pre-trained recognition model based on the mean squared error and text similarity.

[0142] As an example, the parameter can be updated using the following formula: Where Raccuracy is the mean square error between the predicted review results and the labeled review results; Raccuracy is the text similarity between the predicted review results and the labeled review results; λ1 and λ2 are coefficients, and R is a parameter.

[0143] The video quality analysis method provided in this disclosure improves the accuracy of determining the multi-dimensional feature fusion results by encouraging the predicted review results to be close to those of manual annotation and by encouraging text similarity to reach a high similarity.

[0144] In one possible implementation, the above method also includes: Step f1: Obtain the first training data; wherein the first training data includes: training video data, and the first training multi-dimensional feature fusion result corresponding to the training video data.

[0145] The first training data may include the video data to be trained and the first multi-dimensional feature fusion result corresponding to the video data to be trained. The first training data is used to train the neural network model. The first training data can be historically collected first training data, etc., without specific limitations.

[0146] Step f2: Based on the video data to be trained and the fusion result of the first multi-dimensional features to be trained, the neural network model is trained to obtain a pre-trained recognition model.

[0147] After determining the training video data and the fusion result of the first training multi-dimensional features, the training video data and the fusion result of the first training multi-dimensional features can be input into a neural network model to obtain a pre-trained recognition model. The neural network model can be a neural network framework; by training this neural network model, a pre-trained recognition model can be obtained.

[0148] In one possible implementation, the loss function during the training of the neural network model can be: ;in, Indicator numerical prediction loss, Indicate the cause of the loss. For predicted values, For labeling numerical values.

[0149] Step f3: Generalize the first training data to obtain the second training data and the fusion result of the second multi-dimensional features to be trained; wherein the number of second training data is greater than the number of first training data.

[0150] The number of first training data can be multiple. The number of first training data can be 10,000, 11,000, etc. Based on the first training data, a large number of second training data and the fusion results of the second multi-dimensional features to be trained can be further generalized.

[0151] As an example, when dealing with image data, the first training data can be generalized by adjusting brightness, saturation, and other parameters.

[0152] As an example, when dealing with text data, synonyms can be determined based on the first training data to generalize the first training data.

[0153] Step f4: Retrain the pre-trained recognition model using the second training data and the fusion result of the second multi-dimensional features to be trained.

[0154] After obtaining more training data (i.e., the second training data, the second multi-dimensional feature fusion result to be trained), the pre-trained recognition model can be retrained based on the second training data and the second multi-dimensional feature fusion result to be trained.

[0155] The video quality analysis method provided in this disclosure first uses a small amount of high-quality seed data (videos + scores + detailed explanations) for supervised fine-tuning to initialize a model with basic reasoning capabilities. Then, it uses Group Relative Policy Optimization (GRPO) to enhance the model, allowing it to improve and generalize on a larger data space. This enables the model to produce 100,000 or even more results from 10,000 data points, significantly reducing annotation costs.

[0156] In one possible implementation, the above method also includes: Step g1: Obtain the feedback results corresponding to the video data.

[0157] Feedback results can indicate whether the video data meets the user's requirements. The feedback results can be determined by the user. For example, feedback results can indicate that the video data violates regulations.

[0158] Step g2: When the feedback result indicates that the video data does not meet the preset requirements, the pre-trained recognition model is retrained according to the feedback result corresponding to the video data.

[0159] The preset requirements can indicate that the video data is acceptable. When the feedback result indicates that the video data does not meet the preset requirements, it indicates that the video data is unacceptable, and the pre-trained recognition model needs to be retrained based on the feedback result corresponding to the video data.

[0160] The video quality analysis method provided in this disclosure can accurately locate the weak points of the model (such as errors in recognizing specific actions or interference from complex backgrounds) by providing feedback results. During retraining, optimization only needs to be performed on these samples, avoiding the computational waste of retraining with full data.

[0161] This embodiment also provides a video quality analysis device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0162] This embodiment provides a video quality analysis device, such as... Figure 7As shown, the system includes: an extraction module 701, used to extract target data from the video data to be analyzed for quality; wherein the target data includes video frame data, text data, and audio data; a feature extraction processing module 702, used to perform feature extraction processing on the target data to obtain multiple target features corresponding to the target data; a feature fusion processing module 703, used to perform feature fusion processing on the multiple target features to obtain a multi-dimensional feature fusion result; and a determination module 704, used to determine the video quality analysis result corresponding to the video data based on the multi-dimensional feature fusion result.

[0163] In one possible implementation, the feature fusion processing module 703 includes: a merging unit for merging at least two target features with the same target type into a first target feature group; wherein the target type includes video frame type, text type and audio type; and a fusion unit for fusing multiple first target feature groups to obtain a multi-dimensional feature fusion result.

[0164] In one possible implementation, the fusion unit includes: a prediction subunit for predicting a second target feature group that corresponds one-to-one with the first target feature group; wherein the second target feature group is obtained by predicting at least two first target feature groups; and a first determination subunit for determining the multi-dimensional feature fusion result based on the first target feature group and the second target feature group.

[0165] In one possible implementation, the first target feature group is denoted as a first feature capsule, and the second target feature group is denoted as a second feature capsule; and the first determining subunit includes: a mapping subunit, used to map the pose vector of the first feature capsule to the target vector of the second feature capsule based on the pose vector of the first feature capsule; a second determining subunit, used to determine a first coupling coefficient of the second feature capsule based on the first feature capsule; wherein the first coupling coefficient indicates the contribution of the first feature capsule to the second feature capsule; and a third determining subunit, used to determine the multi-dimensional feature fusion result based on the first coupling coefficient and multiple first feature capsules.

[0166] In one possible implementation, the third determining subunit includes: a fourth determining subunit, configured to determine the output value of the second target feature group based on the first coupling coefficient and multiple first feature capsules; wherein the output value indicates the probability of the existence of the second feature capsule; an updating subunit, configured to update the first coupling coefficient based on the output value of the second feature capsule to obtain the second coupling coefficient; and a repeating subunit, configured to repeatedly execute the steps of determining the output value of the second target feature group based on the first coupling coefficient and multiple first feature capsules to updating the first coupling coefficient based on the output value of the second feature capsule to obtain the second coupling coefficient, until the number of repetitions reaches a threshold, thereby determining the multi-dimensional feature fusion result.

[0167] In one possible implementation, the first feature capsule includes: a visual semantic capsule, a visual motion capsule, a textual semantic capsule, a paralinguistic feature capsule, and a title capsule.

[0168] In one possible implementation, the feature extraction processing module 702 includes: a first determining unit, configured to determine keyframes and transition frames based on video frame data using a pre-trained image classification task model; wherein the video frame data is the input of the pre-trained image classification task model, and the keyframes and transition frames are the outputs of the pre-trained image classification task model; a second determining unit, configured to determine the visual semantic features corresponding to the keyframes based on the keyframes using a pre-trained semantic feature extraction model; wherein the keyframes are the input of the pre-trained semantic feature extraction model, and the visual semantic features are the outputs of the pre-trained semantic feature extraction model; and a third determining unit, configured to determine the visual motion features corresponding to the transition frames based on the transition frames using a pre-trained convolutional neural network model; wherein the transition frames are the input of the pre-trained convolutional neural network model, and the visual motion features are the outputs of the pre-trained convolutional neural network model.

[0169] In one possible implementation, the feature extraction processing module 702 includes: a fourth determining unit, used to determine the word-level embedding and sentence-level embedding corresponding to the text data based on the text data using a pre-trained language recognition model; wherein the text data is the input of the pre-trained language recognition model, and the word-level embedding and sentence-level embedding are both the outputs of the pre-trained language recognition model, and the word-level embedding and sentence-level embedding indicate the features corresponding to the text data.

[0170] In one possible implementation, the feature extraction processing module 702 includes: a fifth determining unit, used to perform local feature extraction on the audio data to determine the non-textual data corresponding to the audio data; wherein the non-textual data includes at least: intonation data, speech rate data, and emotional color data; a sixth determining unit, used to use a pre-trained long short-term memory network model to determine the temporal evolution pattern information corresponding to the non-textual data based on the non-textual data; wherein the non-textual data is the input of the pre-trained long short-term memory network model, and the temporal evolution pattern information is the output of the pre-trained long short-term memory network model; and an extraction unit, used to extract the paralinguistic features corresponding to the temporal evolution pattern information based on the temporal evolution pattern information.

[0171] In one possible implementation, the feature fusion processing module 703 is further configured to use a pre-trained recognition model to determine the video quality analysis result corresponding to the video data based on the multi-dimensional feature fusion result; wherein the multi-dimensional feature fusion result is the input of the pre-trained recognition model, and the video quality analysis result is the output of the pre-trained recognition model.

[0172] In one possible implementation, the above apparatus further includes: a review result prediction module, used to input video data into a pre-trained recognition model to obtain a predicted review result corresponding to the video data; wherein the video data is the input of the pre-trained recognition model, and the predicted review result is the output of the pre-trained recognition model; a first determination module, used to determine the mean square error and text similarity based on the predicted review result and the labeled review result; and an update module, used to update the parameters of the pre-trained recognition model based on the mean square error and text similarity.

[0173] In one possible implementation, the above apparatus further includes: a first training data acquisition module, used to acquire first training data; wherein the first training data includes: training video data, and a first training multi-dimensional feature fusion result corresponding to the training video data; a training module, used to train a neural network model based on the training video data and the first training multi-dimensional feature fusion result to obtain a pre-trained recognition model; a generalization processing module, used to perform generalization processing on the first training data to obtain second training data and a second training multi-dimensional feature fusion result; wherein the quantity of the second training data is greater than the quantity of the first training data; and a first retraining module, used to retrain the pre-trained recognition model using the second training data and the second training multi-dimensional feature fusion result.

[0174] In one possible implementation, the above-mentioned device further includes: a feedback result acquisition module, used to acquire the feedback result corresponding to the video data; and a second training module, used to retrain the pre-trained recognition model according to the feedback result corresponding to the video data when the feedback result indicates that the video data does not meet the preset requirements.

[0175] The video quality analysis apparatus provided in this disclosure can execute the video quality analysis method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0176] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0177] The following is a detailed reference. Figure 8The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from memory 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device. The processor 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0178] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0179] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a memory 808, or installed from a ROM 802. When the computer program is executed by the processor 801, it performs the functions defined in the video quality analysis method of embodiments of this disclosure.

[0180] Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0181] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the video quality analysis method shown in the above embodiments is implemented.

[0182] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0183] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A video quality analysis method, characterized in that, The method includes: Extract target data from the video data to be analyzed for quality; wherein, the target data includes video frame data, text data, and audio data; The target data is subjected to feature extraction processing to obtain multiple target features corresponding to the target data; The multiple target features are subjected to feature fusion processing to obtain a multi-dimensional feature fusion result; Based on the multi-dimensional feature fusion results, the video quality analysis results corresponding to the video data are determined.

2. The video quality analysis method according to claim 1, characterized in that, The multiple target features are subjected to feature fusion processing to obtain a multi-dimensional feature fusion result, including: At least two target features with the same target type are merged into a first target feature group; wherein, the target type includes video frame type, text type and audio type; Multiple first target feature groups are fused to obtain the multi-dimensional feature fusion result.

3. The video quality analysis method according to claim 2, characterized in that, The process of fusing multiple first target feature groups to obtain the multi-dimensional feature fusion result includes: Predict a second target feature group that corresponds one-to-one with the first target feature group; wherein the second target feature group is obtained by predicting at least two first target feature groups; The multi-dimensional feature fusion result is determined based on the first target feature group and the second target feature group.

4. The video quality analysis method according to claim 3, characterized in that, The first target feature group is represented as a first feature capsule, and the second target feature group is represented as a second feature capsule; And determining the multi-dimensional feature fusion result based on the first target feature group and the second target feature group includes: Based on the pose vector of the first feature capsule, the pose vector of the first feature capsule is mapped to the target vector of the second feature capsule; Based on the first feature capsule, a first coupling coefficient of the second feature capsule is determined; wherein the first coupling coefficient indicates the contribution of the first feature capsule to the second feature capsule; The multi-dimensional feature fusion result is determined based on the first coupling coefficient and multiple first feature capsules.

5. The video quality analysis method according to claim 4, characterized in that, Based on the first coupling coefficient and multiple first feature capsules, the multi-dimensional feature fusion result is determined, including: Based on the first coupling coefficient and the plurality of first feature capsules, the output value of the second target feature group is determined; wherein the output value indicates the probability of the existence of the second feature capsule; Based on the output value of the second feature capsule, update the first coupling coefficient to obtain the second coupling coefficient; Based on the second coupling coefficient, the steps of determining the output value of the second target feature group based on the first coupling coefficient and multiple first feature capsules, updating the first coupling coefficient based on the output value of the second feature capsule, and obtaining the second coupling coefficient are repeated until the number of repetitions reaches a threshold, and the multi-dimensional feature fusion result is determined.

6. The video quality analysis method according to claim 4, characterized in that, The first feature capsule includes: a visual semantic capsule, a visual motion capsule, a textual semantic capsule, a paralinguistic feature capsule, and a title capsule.

7. A video quality analysis device, characterized in that, The device includes: An extraction module is used to extract target data from video data to be analyzed for quality; wherein, the target data includes video frame data, text data, and audio data; The feature extraction processing module is used to perform feature extraction processing on the target data to obtain multiple target features corresponding to the target data; The feature fusion processing module is used to perform feature fusion processing on the multiple target features to obtain a multi-dimensional feature fusion result. The determination module is used to determine the video quality analysis result corresponding to the video data based on the multi-dimensional feature fusion result.

8. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the video quality analysis method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the video quality analysis method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the video quality analysis method according to any one of claims 1 to 6.

Images