Audio and video audiovisual quality evaluation method and system, electronic equipment and storage medium

Abstract

The invention provides an audio-visual quality evaluation method and system for audios and videos, electronic equipment and a storage medium, and relates to the technical field of audio-video evaluation, and the method comprises the steps: collecting the biometric data of a user in real time in the playing process of a target audio and video, and the environment data of a playing environment; performing feature extraction according to the biometric data to obtain biometric features, and converting the environmental data into environmental features; performing feature extraction on the target audio and video to obtain audio-visual signal features; performing multi-modal feature fusion on the biometric features, the environment features and the audio-visual signal features to obtain a fusion feature vector; and inputting the fusion feature vector into a pre-trained audio-visual quality evaluation model to obtain an audio-visual quality evaluation result of the target audio and video. According to the method, subjective perception and accurate prediction of the audio-visual quality are realized, the limitation that only signal indexes are relied on traditionally is broken through, and the accuracy and individuation level of audio-visual quality evaluation are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of audio and video evaluation technology, specifically to an audio and video audio-visual quality evaluation method, system, electronic device, and storage medium. Background Technology

[0002] Audio and video quality detection and optimization are core aspects of ensuring user experience quality. The main function of audio-visual quality assessment technology is to quantify the degree of distortion of audio and video signals during transmission, thereby accurately assessing the audio-visual quality level perceived by end users.

[0003] Current mainstream audiovisual quality assessment methods are primarily based on objective signal index analysis, including parameters such as peak signal-to-noise ratio and structural similarity. They judge audio and video quality by measuring indicators such as video sharpness and audio distortion. However, these technical solutions focus only on the physical characteristics of the content itself, failing to fully consider individual differences in user subjective perception and the influence of the usage environment. This directly leads to the inaccuracy of existing audiovisual quality assessment methods. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, this application provides a method, system, electronic device and storage medium for audio and video audiovisual quality assessment, which effectively solves the problem of insufficient accuracy of existing audiovisual quality assessment methods.

[0005] In a first aspect, this application provides a method for evaluating the audiovisual quality of audio and video, the method comprising: During the playback of the target audio and video, the user's biometric data and the environmental data of the playback environment are collected in real time. Feature extraction is performed based on the biometric data to obtain biometric features, and the environmental data is transformed into environmental features; Feature extraction is performed on the target audio and video to obtain audiovisual signal features; The biometric features, environmental features, and audiovisual signal features are fused using multimodal features to obtain a fused feature vector; The fused feature vector is input into a pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

[0006] In an optional implementation, the biometric data includes at least heart rate data and facial images, and the environmental data includes at least background noise decibels and light intensity.

[0007] In an optional implementation, when the biometric data is the facial image, the step of extracting features from the biometric data to obtain biometric features includes: The facial image is preprocessed to obtain an initial facial image; Locate the face region of the initial facial image, extract key points of the eye contour within the face region, and obtain an eye region image; Locate the pupil edge of the eye region image, and calculate the geometric center coordinates of the pupil based on the pupil edge; A head pose model is constructed based on the initial facial image. The gaze vector is determined by combining the geometric center coordinates and the key points of the eye contour. The gaze vector is then projected onto a two-dimensional screen coordinate system to obtain the gaze point coordinates. The effective fixation duration is calculated based on the coordinates of the fixation point and the effective fixation area of ​​the target audio and video to obtain eye movement features.

[0008] In an optional implementation, the step of extracting features from the target audio and video to obtain audiovisual signal features includes: Extract the video keyframes of the target audio and video, and convert the video keyframes into grayscale images; Perform Laplacian convolution on the grayscale image to obtain a convolutional image; Calculate the variance of all pixel values ​​in the convolutional image to obtain sharpness features; The target audio and video streams are segmented and windowed to obtain windowed audio frames; Perform a Fast Fourier Transform on the windowed audio frame to obtain a complex spectrum; The power spectral density is calculated based on the complex spectrum, and the spectrum is divided into sub-bands based on the power spectral density and the logarithmic energy is calculated to obtain the frequency band energy characteristics.

[0009] In an optional implementation, the step of fusing the biometric features, the environmental features, and the audiovisual signal features into a multimodal feature vector to obtain a fused feature vector includes: The biometric features, environmental features, and audiovisual signal features are weighted and fused based on the correlation matrix to obtain a first fused feature vector. And / or based on vector embedding, interactive modeling is performed on the biometric features, the environmental features, and the audiovisual signal features to obtain a second fused feature vector.

[0010] In an optional implementation, the audiovisual quality assessment model employs a two-layer neural network architecture, wherein: The first layer of the neural network structure processes static frequency domain features, while the second layer processes dynamic temporal features.

[0011] In an optional implementation, the method further includes: Based on the audiovisual quality assessment results, the playback parameters of the target audio and video are adjusted, and a prompt message is sent to the user.

[0012] Secondly, this application provides an audio-visual quality assessment system, the system comprising: The data acquisition module is used to collect the user's biometric data and the environmental data of the playback environment in real time during the playback of the target audio and video. The first extraction module is used to extract features based on the biometric data to obtain biometric features, and to convert the environmental data into environmental features. The second extraction module is used to extract features from the target audio and video to obtain audiovisual signal features; The feature fusion module is used to perform multimodal feature fusion of the biometric features, the environmental features, and the audiovisual signal features to obtain a fused feature vector; The quality assessment module is used to input the fused feature vector into a pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

[0013] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the audio-visual quality assessment method for audio and video as described in any of the foregoing embodiments.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the audio-visual quality assessment method for audio and video as described in any of the foregoing embodiments.

[0015] The audio-visual quality assessment method, system, electronic device, and storage medium provided in this application, by integrating user biometric data and environmental parameters to construct a multimodal machine learning model, achieve accurate prediction of subjective perception of audio-visual quality, breaking through the limitations of traditional methods that rely solely on signal indicators. It significantly improves the accuracy and personalization of audio-visual quality assessment, dynamically identifies environmental interference and changes in user attention, and outputs optimization instructions in real time, thereby effectively improving user experience. It also possesses high practicality and scalability, and can be widely applied to smart terminals and streaming media platforms. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a first schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application; Figure 2 This is a second schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application; Figure 3 This is a third schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application; Figure 4 This is the fourth schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the audio-visual quality assessment system provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0018] Explanation of key component symbols: 200. Audio-visual quality assessment system; 210. Data acquisition module; 220. First extraction module; 230. Second extraction module; 240. Feature fusion module; 250. Quality assessment module; 300. Electronic device; 310. Processor; 320. Communication interface; 330. Memory; 340. Communication bus. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be further described clearly and completely below with reference to the accompanying drawings of the embodiments. It should be noted that the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0022] Current mainstream audiovisual quality assessment methods are primarily based on objective signal index analysis, including parameters such as peak signal-to-noise ratio (PSNR) and structural similarity. Audiovisual quality is judged by measuring metrics such as video sharpness and audio distortion. However, in practical applications, users' perception of audiovisual quality is affected by physiological states and environmental interference. For example, even with a perfect signal in a noisy environment, users may perceive poor audio, or the video may appear blurry under strong light, but objective indicators cannot reflect this deficiency. Therefore, existing audiovisual quality assessment methods suffer from insufficient accuracy.

[0023] Example 1 This application provides a method for evaluating the audiovisual quality of audio and video, effectively solving the problem of insufficient accuracy in existing audiovisual quality evaluation methods. Figure 1 This is a first schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application, as shown below. Figure 1 As shown, the method includes the following steps: S100: Collects user biometric data and environmental data of the playback environment in real time during the playback of the target audio and video.

[0024] In this embodiment, the target audio or video can be video or audio, and the target audio or video is played on the user's electronic device, which includes, but is not limited to, devices such as mobile phones and smart TVs, and is equipped with integrated sensors such as cameras, microphones, wearable heart rate monitors and light sensors.

[0025] In this embodiment, biometric data and environmental data are continuously collected during the playback of the target audio / video. Biometric data includes, but is not limited to, heart rate data and facial images. Heart rate data can be monitored using a wearable heart rate monitor, with a collection frequency set to once per second. Facial images are captured by a camera at a fixed frame rate, which can be set to 30 FPS. Environmental data includes, but is not limited to, background noise levels in decibels and light intensity. The background noise level in decibels is measured using a microphone, and the light intensity in the playback environment is detected using a light sensor.

[0026] S200. Based on biometric data, feature extraction is performed to obtain biometric features, and environmental data is transformed into environmental features.

[0027] As an optional implementation of this application, the three types of data—heart rate data, background noise decibel value, and light intensity—are all in numerical form. Therefore, the collected dataset can be directly processed by data cleaning and data standardization to obtain the corresponding heart rate features, background noise decibel features, and light intensity features.

[0028] For facial images, image processing techniques are used to locate the pupil center and corner of the eye feature points. Combined with head posture estimation, the cumulative time that the gaze falls on the target area of ​​the target audio and video playback screen is calculated to obtain eye movement features. Figure 2 This is a second schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application, as shown below. Figure 2 As shown, feature extraction from facial images specifically includes the following steps: S210. Perform image preprocessing on the facial image to obtain an initial facial image.

[0029] Optionally, the acquired facial image can be converted to grayscale to reduce computation, highlight brightness information, and facilitate subsequent feature extraction. Simultaneously, Gaussian filtering can be used to smooth the image by weighted averaging of pixels, effectively suppressing high-frequency noise, such as abrupt changes in illumination or sensor noise, avoiding pupil positioning errors caused by light spots, and improving the robustness of eye-tracking detection.

[0030] S220. Locate the face region of the initial facial image, extract key points of the eye contour within the face region, and obtain the eye region image.

[0031] As an optional implementation of this application, a cascaded classifier or deep learning detector can be used to locate the face region in the initial facial image. For example, the Haar Cascades model can be used to quickly eliminate non-face regions through a series of weak classifiers, achieving efficient face detection, suitable for resource-constrained client devices. Alternatively, a multi-task convolutional neural network model can be used to simultaneously perform face localization, key point detection, and pose estimation, suitable for complex backgrounds, occlusions, or small faces.

[0032] Within the located face region, high-precision facial landmark detection algorithms can be used to further analyze facial details and extract key points of the eye contour. After obtaining the key points of the eye contour, the eye region image is cropped based on the position of each key point to reduce the processing range and improve efficiency. For example, Dlib's 68-point model can be used to fit the facial contour and the position of facial features using a regression forest method, where points 37–42 correspond to the left eye and points 43–48 correspond to the right eye, extracting 6 key points for each eye to accurately mark the positions of the upper and lower eyelids and the corners of the eyes. Alternatively, the MediaPipe Face Mesh model based on deep learning can be used to output 468 facial mesh points in 3D space, from which dense key points around the left and right eyes can be selected, with 6–8 core points selected for each eye, achieving more robust eye contour modeling.

[0033] S230. Locate the pupil edge of the eye region image and calculate the geometric center coordinates of the pupil based on the pupil edge.

[0034] Optionally, a binarization thresholding process is applied to the eye region image. By setting an appropriate threshold, the image is divided into two parts: the darker pupil area is black, and the bright sclera and white of the eye are white, thus highlighting the pupil target. Then, a contour-finding algorithm is used to detect all closed contours, and the contour most likely to represent the pupil is selected based on geometric features such as area and roundness. Finally, image moments are applied to this contour to calculate its centroid, obtaining the geometric center coordinates of the pupil.

[0035] S240. Construct a head pose model based on the initial facial image, determine the gaze vector by combining the geometric center coordinates and key points of the eye contour, and project the gaze vector onto the two-dimensional screen coordinate system to obtain the gaze point coordinates.

[0036] As an optional implementation of this application, to improve gaze tracking accuracy, a PnP algorithm can be used to construct a head pose model. This model utilizes known 3D facial key points and corresponding 2D projection points in the image to calculate the head's rotation angle in three-dimensional space, correcting gaze deviations caused by head movement. Then, the relative displacement vector between the detected geometric center coordinates of the pupil and the key points of the eye contour is used as the gaze vector, reflecting the direction of eyeball rotation within the eye socket. Finally, the gaze vector is input to a pre-established mapping matrix calibrated by the user. This mapping matrix correlates changes in eye features with the actual gaze point on the playback screen. The gaze vector is then projected onto a two-dimensional screen coordinate system using this mapping matrix to obtain the final gaze point coordinates.

[0037] S250. Calculate the effective fixation duration based on the coordinates of the fixation point and the effective fixation area of ​​the target audio and video to obtain eye movement characteristics.

[0038] In this embodiment, the screen playback area of ​​the target audio / video can be set as the effective gaze area. If the coordinates of the gaze point fall within the effective gaze area and remain continuously within it... N A frame is then marked as a valid gaze, where N The value can be set according to the actual situation.

[0039] Eye movement characteristics are obtained by calculating the effective fixation duration. The formula for calculating the effective fixation duration is as follows:

[0040] In the above formula, T gaze Indicates effective fixation duration. Indicates the frame interval. F valid This represents a Boolean value, which is the valid gaze flag for each frame. It takes the value 1 or 0, where 1 indicates a valid gaze and 0 indicates an invalid gaze.

[0041] This application embodiment improves the accuracy and robustness of gaze tracking by integrating key points of the corner of the eye contour, pupil center coordinates, and head posture information. It effectively eliminates positioning deviations caused by head movements, accurately quantifies the user's attention duration on screen content, truly reflects viewing attention, provides objective and continuous behavioral evidence for audiovisual quality assessment, and enhances applicability and reliability in natural interaction scenarios.

[0042] S300: Extract features from the target audio and video to obtain audiovisual signal features.

[0043] In this embodiment, the video and audio signals of the target audio / video are processed separately to obtain audiovisual signal features. Edge detection algorithms can be used to extract the sharpness features of video frames and quantify the blurriness or detail richness of the video image. Fourier transform is used to extract the spectral features of the audio, converting the time-domain sound signal into frequency-domain features and analyzing the frequency distribution. Figure 3 This is a third schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application, as shown in the figure. Figure 3 As shown, the feature extraction of the target audio and video specifically includes the following steps: S310. Extract the video keyframes of the target audio and video, and convert the video keyframes into grayscale images.

[0044] In this embodiment, one or more video keyframes are extracted from the target audio and video every second, and the video keyframes are converted into grayscale images to eliminate color interference, highlight spatial details, and reduce computational load.

[0045] S320. Perform Laplacian convolution on the grayscale image to obtain the convolutional image.

[0046] As an optional implementation of this application, a Laplacian convolution can be performed on the grayscale image. The Laplacian operator calculates the second derivative of the image's grayscale value through convolution operations, exhibiting a high response to regions with drastic changes around pixels, effectively highlighting edges and details. For example, a 3×3 convolution kernel can be slid across the grayscale image, with negative weights at the center and positive weights in the neighborhood, making the response of smooth regions close to zero, while producing obvious peaks at the edges.

[0047] S330. Calculate the variance of all pixel values ​​in the convolutional image to obtain sharpness features.

[0048] In this embodiment, the variance of all pixel values ​​in the convolutional image is calculated to quantify the sharpness of the video frame: a sharp image has sharp edges and a strong Laplacian response, resulting in a large variance of pixel values; a blurry image has smooth edges and a weak Laplacian response, resulting in a small variance of pixel values. Therefore, the variance of all pixel values ​​is used as an objective sharpness feature scalar to assist in evaluating the user's perceived quality.

[0049] S340. Perform frame segmentation and windowing processing on the audio stream of the target audio / video to obtain windowed audio frames.

[0050] In this embodiment, the continuous audio stream is framed in 20ms increments, with a 10ms overlap between frames (i.e., a 10ms frame shift) to ensure temporal continuity and capture dynamic changes. Since directly truncating short frames can cause signal abrupt changes and spectral leakage, each frame is multiplied by a Hamming window function to smoothly attenuate the signal at both ends to zero, reducing boundary discontinuities. Framing and windowing effectively concentrate energy on the main lobe, reducing sidelobe interference and improving the accuracy of frequency domain analysis.

[0051] S350. Perform a Fast Fourier Transform on the windowed audio frame to obtain the complex spectrum.

[0052] In this embodiment, a K-point Fast Fourier Transform (FFT) is performed on each windowed audio frame to convert it from the time domain to the frequency domain, obtaining a complex spectrum containing amplitude and phase information. K can be 512 to provide sufficient frequency resolution, balancing computational efficiency and analytical accuracy. The FFT decomposes the 20ms short-time signal into the energy distribution of different frequency components, reflecting the spectral structure of the audio.

[0053] S360. Calculate the power spectral density based on the complex spectrum, divide the spectrum into sub-bands based on the power spectral density and calculate the logarithmic energy to obtain the frequency band energy characteristics.

[0054] In this embodiment, the power spectral density is obtained by calculating the square of the amplitude of the complex spectrum, which reflects the energy distribution of the signal at various frequencies and is the core of frequency domain analysis. Then, the entire frequency band is divided into several sub-bands, for example, 0-500Hz for low frequency, 500-2000Hz for mid frequency, and above 2000Hz for high frequency. The power spectral density of all frequency points within each sub-band is accumulated, and the logarithm is taken to obtain the frequency band energy characteristics. Finally, frequency band energy feature vectors for low frequency, mid frequency, and high frequency are formed, serving as important indicators of audio quality.

[0055] This application's embodiments effectively quantify key dimensions of user-perceived quality by extracting features from the target audio and video. The video stream employs an edge detection algorithm to sensitively reflect the degree of image blur and capture changes in sharpness. The audio stream uses Fourier transform to extract spectral features, which can identify noise interference and distortion distribution, reflecting the quality of the auditory experience. Simultaneously, it provides reliable and interpretable underlying features for the audiovisual quality assessment model, significantly improving the accuracy and robustness of quality prediction.

[0056] S400: Multimodal feature fusion of biometric features, environmental features, and audiovisual signal features to obtain a fused feature vector.

[0057] As an optional implementation of this application, a fused feature vector can be obtained by weighted fusion of biometric features, environmental features and audiovisual signal features based on a correlation matrix, and / or by interactive modeling of biometric features, environmental features and audiovisual signal features based on vector embedding.

[0058] For example, feature association is performed on heart rate features and background noise decibel features based on a correlation matrix, and the fused output is a high-dimensional feature vector. First, the heart rate feature sequence and the background noise decibel feature sequence are time-aligned, and upsampling is used to make their frequencies consistent. Z-score normalization is then used to eliminate dimensional differences, resulting in heart rate feature vectors and noise feature vectors. Then, within a fixed time window, the Pearson correlation coefficient or covariance of the heart rate feature vector and the noise feature vector is calculated to generate a correlation scalar, reflecting the dynamic correlation strength between the two. This correlation scalar is then introduced into an attention mechanism as a modulator. The heart rate feature and background noise decibel features are concatenated and input into a fully connected layer, where the bias term or weight gating is modulated by the correlation scalar. When noise is high and heart rate variability is enhanced (i.e., when the correlation is strong), the contribution weight of the noise feature to the audiovisual quality degradation is automatically indicated, obtaining the attention weights corresponding to the two features. Finally, the two features are weighted and fused according to the attention weights to obtain the high-dimensional feature vector.

[0059] For example, an interactive modeling approach is used to model eye-tracking features and sharpness features based on vector embedding, fusing the output into another high-dimensional feature vector. First, continuous eye-tracking features are discretized into multiple levels and mapped to dense vectors through an embedding layer to obtain the eye-tracking feature vector. Simultaneously, sharpness features are mapped to vectors of the same dimension to obtain the sharpness feature vector. Then, dot product or Hadamard product is used to calculate the interaction features between the eye-tracking feature vector and the sharpness feature vector to obtain a high-dimensional feature vector, used to capture the nonlinear effects when low sharpness and low fixation occur simultaneously.

[0060] It is understandable that the above two methods can be used to perform multimodal feature fusion on various biometric features, environmental features, and audiovisual signal features to obtain different fused feature vectors. This effectively captures the correlation between user subjective perception, environmental interference, and audiovisual features, improves the accuracy and robustness of quality prediction, and enhances the model's adaptability to complex real-world scenarios.

[0061] S500: Input the fused feature vector into the pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

[0062] In the embodiments of this application, the audiovisual quality assessment model can adopt a multimodal neural network model, which is a two-layer neural network architecture. The first layer of the neural network structure processes static frequency domain features, and the second layer of the neural network structure processes dynamic temporal features.

[0063] As an optional implementation of this application, the first layer neural network structure can adopt a convolutional neural network model, which is used to process static frequency domain features such as sharpness features, brightness features, and ambient lighting features. For example, this convolutional neural network model includes three fully connected layers, uses ReLU as the activation function, and includes a Dropout layer to prevent overfitting.

[0064] Optionally, the second-layer neural network structure can employ a recurrent neural network model to process dynamic temporal features such as heart rate characteristics, eye movement characteristics, and background noise decibel characteristics. For example, this recurrent neural network model can employ two layers of gated recurrent units to capture time dependencies.

[0065] As an optional implementation of this application, the output layer of the audiovisual quality assessment model includes a regression head and a classification head. The regression head includes one neuron, uses the sigmoid activation function, outputs a result of 0-1, and maps it to a score of 0-100. The classification head includes three neurons, uses the softmax activation function, and outputs three probabilities: high, medium, and low.

[0066] As an optional implementation of this application, a labeled dataset is used as the training dataset for the model, including subjective ratings, synchronous biometric data, and environmental data. The training dataset has at least 1000 samples. Random Gaussian noise is added to the training dataset to simulate sensor errors, and the training dataset is divided into training, testing, and validation sets for multi-strategy joint optimization training, thereby improving model performance and robustness.

[0067] During training, the loss function of the audiovisual quality assessment model adopts the joint loss function, the expression of which is as follows:

[0068] In the above formula, L total Represents the joint loss value. and These represent the corresponding weighting factors of the loss term. This represents the mean squared error loss term, used to supervise the regression output and ensure that the predicted score closely approximates the user's subjective average opinion score. This represents the cross-entropy loss term, used to supervise the classification output and enhance the ability to distinguish the output category.

[0069] Optionally, the optimizer uses the Adam optimizer with an initial learning rate of 0.0001. A learning rate decay strategy is employed, automatically decreasing the learning rate when the validation loss stagnates to avoid getting trapped in local optima. Five-fold cross-validation is used to evaluate the model's generalization performance, reducing the risk of overfitting due to data distribution bias. An early stopping mechanism is also enabled: if the total validation loss does not decrease for 10 consecutive epochs, training is terminated early to save computational resources and prevent overfitting.

[0070] In this embodiment of the application, the fused feature vector is input into the trained audiovisual quality assessment model for evaluation to obtain the audiovisual quality assessment result of the target audio and video. The audiovisual quality assessment result includes an audiovisual quality assessment score of 0-100 points and a classification result of high, medium and low.

[0071] As an optional implementation of this application, the output layer of the audiovisual quality assessment model can be improved by setting auxiliary output nodes to predict and obtain video quality scores and audio environment scores respectively.

[0072] As a further implementation of the embodiments of this application, Figure 4 This is the fourth schematic diagram of the audio-visual quality assessment method provided in the embodiments of this application, as shown below. Figure 4 As shown, the method also includes the following steps: S600 adjusts the playback parameters of the target audio and video based on the audio-visual quality assessment results and sends a prompt message to the user.

[0073] In this embodiment of the application, if the audiovisual quality assessment score is lower than the score threshold, an adjustment suggestion is generated based on the model prediction result and sent to the electronic device playing the target audio and video to adjust the playback parameters of the target audio and video, optimize the playback parameters in real time, send prompt information to the user, record user feedback to iterate the model, and the feedback format can be JSON instructions.

[0074] For example, the score threshold is set to 70 points. When the audiovisual quality assessment score is less than 70 points, the video quality score is checked to see if it is lower than the threshold. If it is lower than the threshold, instructions such as adjusting resolution and brightness are generated. At the same time, the audio environment score is checked to see if it is lower than the threshold. If it is lower than the threshold, instructions such as enabling noise reduction and volume enhancement are generated.

[0075] As an optional implementation of this application, rule-based post-processing can be used to quickly diagnose problems such as environmental interference, visual fatigue, and transmission quality after the audiovisual quality assessment score is found to be lower than the score threshold. Feedback instructions can then be generated for these problems.

[0076] For example, if the background noise level in the playback environment is greater than 60 dB and the heart rate changes too quickly, it is determined to be noise interference; if the light intensity in the playback environment is less than 50 lux and the eye movement feature value decreases, it is determined to be insufficient light or unclear content; if the Laplace variance is consistently too low, it is determined to be insufficient resolution.

[0077] This application's embodiments predict the actual audio and video content quality perceived by the user by collecting the user's biometric data and playback environment data. This data is then combined with audiovisual signal characteristics and input into a prediction model to capture the user's actual reactions and external interference, directly improving the accuracy and personalization of audiovisual instruction evaluation. This overcomes the limitations of traditional objective indicators and can help streaming media platforms or smart devices optimize content in real time, enhancing the user experience.

[0078] Example 2 Based on the same technical concept as Embodiment 1 above, this application provides an audio-visual quality assessment system. Figure 5 This is a schematic diagram of the audio-visual quality assessment system provided in the embodiments of this application, as shown below. Figure 5 As shown, the audio-visual quality assessment system 200 includes: The data acquisition module 210 is used to collect the user's biometric data and the environmental data of the playback environment in real time during the playback of the target audio and video.

[0079] The first extraction module 220 is used to extract features from biometric data, obtain biometric features, and transform environmental data into environmental features.

[0080] The second extraction module 230 is used to extract features from the target audio and video to obtain audiovisual signal features.

[0081] The feature fusion module 240 is used to perform multimodal feature fusion of biometric features, environmental features and audiovisual signal features to obtain a fused feature vector.

[0082] The quality assessment module 250 is used to input the fused feature vector into a pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

[0083] The audio-visual quality assessment system provided in this application integrates user biometric data and environmental parameters to achieve accurate prediction of subjective perception of audio-visual quality, thereby significantly improving the accuracy and personalization of audio-visual quality assessment.

[0084] It is understood that the implementation method of the audio-visual quality assessment method in Embodiment 1 above is also applicable to this embodiment and can achieve the same technical effect, so it will not be described again here.

[0085] Example 3 Based on the same concept, this application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the electronic device 300 may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute the steps of the audio-visual quality assessment method as described in the above embodiments. For example, this includes: S100: Real-time collection of user biometric data and environmental data of the playback environment during the playback of the target audio and video; S200. Based on biometric data, feature extraction is performed to obtain biometric features, and environmental data is transformed into environmental features. S300: Extract features from the target audio and video to obtain audiovisual signal features; S400: Multimodal feature fusion of biometric features, environmental features, and audiovisual signal features to obtain a fused feature vector; S500: Input the fused feature vector into the pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

[0086] The processor 310 can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0087] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The memory 330 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] Example 4 Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program containing at least one piece of code executable by a master control device to control the master control device to implement the audio-visual quality assessment methods as described in the above embodiments. For example, it includes: S100: Real-time collection of user biometric data and environmental data of the playback environment during the playback of the target audio and video; S200. Based on biometric data, feature extraction is performed to obtain biometric features, and environmental data is transformed into environmental features. S300: Extract features from the target audio and video to obtain audiovisual signal features; S400: Multimodal feature fusion of biometric features, environmental features, and audiovisual signal features to obtain a fused feature vector; S500: Input the fused feature vector into the pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

[0090] Based on the same technical concept, this application also provides a computer program, which, when executed by a main control device, is used to implement the above-described method embodiments.

[0091] The computer program may be stored, in whole or in part, on a computer-readable storage medium packaged with the processor, or in part or in whole on a memory not packaged with the processor.

[0092] Based on the same technical concept, this application also provides a processor for implementing the above-described method embodiments. The processor can be a chip.

[0093] In summary, the audio-visual quality assessment method, system, electronic device, and storage medium provided in this application, by integrating user biometric data and environmental parameters to construct a multimodal machine learning model, achieve accurate prediction of subjective perception of audio-visual quality, overcoming the limitations of traditional methods that rely solely on signal indicators. This significantly improves the accuracy and personalization of audio-visual quality assessment, dynamically identifies environmental interference and changes in user attention, and outputs optimization instructions in real time, thereby effectively improving user experience. Furthermore, it possesses high practicality and scalability, and can be widely applied to smart terminals and streaming media platforms.

[0094] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0095] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for evaluating the audiovisual quality of audio and video, characterized in that, The method includes: During the playback of the target audio and video, the user's biometric data and the environmental data of the playback environment are collected in real time. Feature extraction is performed based on the biometric data to obtain biometric features, and the environmental data is transformed into environmental features; Feature extraction is performed on the target audio and video to obtain audiovisual signal features; The biometric features, environmental features, and audiovisual signal features are fused using multimodal features to obtain a fused feature vector; The fused feature vector is input into a pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

2. The audio-visual quality assessment method according to claim 1, characterized in that, The biometric data includes at least heart rate data and facial images, and the environmental data includes at least background noise decibels and light intensity.

3. The audio-visual quality assessment method according to claim 2, characterized in that, When the biometric data is the facial image, the step of extracting features from the biometric data to obtain biometric features includes: The facial image is preprocessed to obtain an initial facial image; Locate the face region of the initial facial image, extract key points of the eye contour within the face region, and obtain an eye region image; Locate the pupil edge of the eye region image, and calculate the geometric center coordinates of the pupil based on the pupil edge; A head pose model is constructed based on the initial facial image. The gaze vector is determined by combining the geometric center coordinates and the key points of the eye contour. The gaze vector is then projected onto a two-dimensional screen coordinate system to obtain the gaze point coordinates. The effective fixation duration is calculated based on the coordinates of the fixation point and the effective fixation area of ​​the target audio and video to obtain eye movement features.

4. The audio-visual quality assessment method according to claim 1, characterized in that, The step of extracting features from the target audio and video to obtain audiovisual signal features includes: Extract the video keyframes of the target audio and video, and convert the video keyframes into grayscale images; Perform Laplacian convolution on the grayscale image to obtain a convolutional image; Calculate the variance of all pixel values ​​in the convolutional image to obtain sharpness features; The target audio and video streams are segmented and windowed to obtain windowed audio frames; Perform a Fast Fourier Transform on the windowed audio frame to obtain a complex spectrum; The power spectral density is calculated based on the complex spectrum, and the spectrum is divided into sub-bands based on the power spectral density and the logarithmic energy is calculated to obtain the frequency band energy characteristics.

5. The audiovisual quality assessment method according to claim 1, characterized in that, The step of fusing the biometric features, environmental features, and audiovisual signal features into a multimodal feature vector to obtain a fused feature vector includes: The biometric features, environmental features, and audiovisual signal features are weighted and fused based on the correlation matrix to obtain a first fused feature vector. And / or based on vector embedding, interactive modeling is performed on the biometric features, the environmental features, and the audiovisual signal features to obtain a second fused feature vector.

6. The audio-visual quality assessment method according to claim 1, characterized in that, The audiovisual quality assessment model adopts a two-layer neural network architecture, wherein: The first layer of the neural network structure processes static frequency domain features, while the second layer processes dynamic temporal features.

7. The audio-visual quality assessment method according to claim 1, characterized in that, The method further includes: Based on the audiovisual quality assessment results, the playback parameters of the target audio and video are adjusted, and a prompt message is sent to the user.

8. An audio-visual quality assessment system, characterized in that, The system includes: The data acquisition module is used to collect the user's biometric data and the environmental data of the playback environment in real time during the playback of the target audio and video. The first extraction module is used to extract features based on the biometric data to obtain biometric features, and to convert the environmental data into environmental features. The second extraction module is used to extract features from the target audio and video to obtain audiovisual signal features; The feature fusion module is used to perform multimodal feature fusion of the biometric features, the environmental features, and the audiovisual signal features to obtain a fused feature vector; The quality assessment module is used to input the fused feature vector into a pre-trained audiovisual quality assessment model to obtain the audiovisual quality assessment results of the target audio and video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the audio-visual quality assessment method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the audiovisual quality assessment method for audio and video as described in any one of claims 1-7.

Images