Video quality evaluation method, device, equipment, medium and product

Abstract

The invention relates to a video quality evaluation method and device, equipment, a medium and a product. The method comprises the following steps: acquiring operation characteristics in a target video and the complexity of the target video; fusing the complexity and the operation feature to obtain a fusion feature of the target video; performing quality evaluation on the fusion features through a quality evaluation model to obtain a quality score of the target video; the quality score is used for representing the fuzzy degree of the target video. According to the scheme, the influence of the complexity of the target video on quality scoring is considered, and the accuracy of quality scoring is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to video quality evaluation methods, devices, equipment, media and products. Background Art

[0002] During the video transmission process, due to the influence of data transmission and various video processing algorithms, the video quality will be degraded compared to the original video, resulting in potential problems in subsequent use and also having a certain impact on people's visual experience. Therefore, the evaluation of video quality is very important.

[0003] There are generally two solutions in the related art, one is to evaluate video quality based on reference video, and the other is to evaluate video quality without reference video. In practice, since there is no reference video available in many cases, video quality evaluation without reference video is more widely used.

[0004] For the evaluation of video quality without reference video, the related technical solution is generally to directly extract video features and perform video quality evaluation based on the video features. Since the quality of the video may be affected by various factors, the accuracy of the quality score obtained by this solution is relatively low. Summary of the invention

[0005] The present application provides a video quality evaluation method, apparatus, device, medium and product. The solution takes into account the influence of the complexity of the target video on the quality score and improves the accuracy of the quality score.

[0006] The technical solutions adopted in this application are as follows:

[0007] In a first aspect, the present application provides a video quality evaluation method, the method comprising: obtaining operating features in a target video and the complexity of the target video; fusing the complexity and operating features to obtain fused features of the target video; performing quality evaluation on the fused features through a quality evaluation model to obtain a quality score of the target video; the quality score is used to characterize the degree of blurriness of the target video.

[0008] Based on the above means, when scoring the quality of the target video, the complexity and motion features of the target video are fused, and the quality of the target video is scored based on the fused features. In practice, different videos may have different corresponding complexities. This solution takes into account the impact of the complexity of the target video on the quality score and improves the accuracy of the quality score.

[0009] In some implementations, obtaining motion features in a target video includes: inputting the target video into a three-dimensional convolution layer, processing the target video through the three-dimensional convolution layer to obtain a first output; the three-dimensional convolution layer is used for temporal feature extraction; the first output is used to characterize the temporal features of the target video; inputting the first output into a two-dimensional convolution layer, processing the first output through the two-dimensional convolution layer to obtain a second output; the two-dimensional convolution layer is used for spatial feature extraction; the second output is used to characterize the temporal features and spatial features of the target video; and determining the motion features based on the second output.

[0010] Based on the above means, when determining motion features, first extract the time features through the three-dimensional convolution layer, and then extract the spatial features based on the two-dimensional convolution layer. The second output obtained in this way can characterize the spatiotemporal features (time features and spatial features) of the target video. In this way, on the one hand, taking the spatiotemporal features as motion features, both the video quality in the time dimension and the video quality in the spatial dimension are considered, so the information of the determined motion features is more comprehensive, so that the quality score of the determined target video is more accurate; on the other hand, the three-dimensional convolution layer and the two-dimensional convolution layer are used to extract the time features and the spatial features respectively. Since the three-dimensional convolution layer and the two-dimensional convolution layer do not require high computing power, the implementation process has low hardware requirements for the device and has a wide range of applications.

[0011] In some implementations, determining the operation characteristic based on the second output includes: reducing the size of the second output to obtain a first size characteristic; superimposing the first size characteristic with the second output to obtain a first intermediate characteristic; determining the first intermediate characteristic as a motion characteristic, or continuing to reduce the size based on the first intermediate characteristic to obtain a motion characteristic.

[0012] Based on the above means, when determining the motion feature, the second output can be further processed. During the processing, the image of the second output can be reduced in size, and then the reduced size feature and the second output are superimposed to obtain the first intermediate feature, and the motion feature is determined based on the first intermediate feature. Since the first intermediate feature is the superposition of the reduced size feature and the second output, the reduced feature and the feature information of the second output can be characterized, so that the motion feature can be highlighted in the overall (second output) and details (reduction); if the first intermediate feature is further reduced in size and cycled several times, detail features of different size dimensions and overall features can be obtained; so that the motion feature can characterize the overall feature and detail features of various size dimensions, so that the quality score determined in this way takes into account the overall quality and the detail quality of various sizes, and the quality score obtained is more accurate.

[0013] In some implementations, obtaining the complexity of the target video includes: determining a standard deviation of each frame of the target video; traversing all images in the target video to obtain standard deviations of multiple images; determining a maximum value of the standard deviations of the multiple images as the spatial complexity of the target video;

[0014] And / or, based on the target video, determine the correlation standard deviation between each frame image and the previous frame image at corresponding pixels to obtain N correlation standard deviations; and determine the maximum value of the N correlation standard deviations as the time complexity of the target video.

[0015] Based on the above means, the spatial complexity and / or temporal complexity can be determined based on the standard deviation. On the one hand, the complexity of the target video can be reflected in different dimensions, and the complexity can be determined more accurately. For example, some video images are more complex, and the corresponding spatial complexity is higher. Some video images are simple, but the images change more, and the corresponding temporal complexity is higher. In the application, the complexity can be determined based on actual needs. It can be seen that the complexity determined by this method is more accurate. On the other hand, determining the complexity by the standard deviation method makes the implementation logic simple and reliable.

[0016] In some implementations, the method further includes: determining that the target video is blurred when the quality score is less than a score threshold; outputting the cause of blurring of the target video through a quality evaluation model; the causes of blurring include: physical factor blurring and algorithmic factor blurring.

[0017] Based on the above means, the reasons for the video module (physical factor blur and algorithmic factor blur) can also be output through the quality evaluation model. In this way, the user can know the reason for the blur of the target video in time, and the user experience is high.

[0018] In some implementations, when the quality score is less than a scoring threshold, the target video is determined to be blurred, including: when the quality score is less than a first scoring threshold, determining that the target video is blurred, and the cause of the blur is algorithmic factor blur; when the quality score is less than a second scoring threshold and greater than a second scoring starting value, determining that the target video is blurred, and the cause of the blur is physical factor blur; wherein the first scoring threshold is less than the second scoring threshold, and the second scoring starting value is greater than the first scoring threshold.

[0019] Based on the above means, since there are two fuzzy factors, the corresponding scores have two scoring ranges. In the first scoring range, if the quality score is less than the first scoring threshold, it is determined that the target video is fuzzy due to algorithmic factors; in the second scoring range, if the quality score is less than the second scoring threshold, it is determined that the target video is fuzzy due to physical factors. The starting value of the second scoring range is the second scoring starting value. In this way, the fuzzy factor of the target video can be determined from the two fuzzy factors through a single scoring model, which is simple, convenient and reliable.

[0020] In some implementations, when the method is applied to vehicle equipment, the method also includes: when the quality score is less than a score threshold, outputting a first reminder message; the first reminder message is used to remind that the current video is blurred to cancel a first function of the vehicle; the first function includes: intelligent driving function, assisted driving function or intelligent reversing function.

[0021] Based on the above means, a reminder message can also be output when the quality score is low. In this way, users can promptly perceive which functions may be affected by the blurry video, and thus cancel the corresponding functions to improve the safety of the vehicle. For example, when the video is blurry, the intelligent driving function can be canceled in time to improve driving safety.

[0022] In some implementations, the method also includes: obtaining training data; the training data includes: multiple training videos, and a subjective quality score of each of the multiple training videos; a type label of each of the multiple training videos, the type labels including clarity, blur caused by physical factors, and blur caused by algorithmic factors; training a convolutional neural network using the training data to obtain a quality evaluation model.

[0023] Based on the above means, the convolutional neural network model can be trained with multiple training data and the subjective quality score of each training video to obtain a quality evaluation model. The quality evaluation model obtained in this way can be used to score videos. Furthermore, the convolutional neural network model can be trained with multiple training data and the subjective quality score of each training video, as well as the type label of each video, to obtain a quality evaluation model. The quality evaluation model obtained in this way can be used to score videos and can also output blur reasons.

[0024] In some implementations, when the method is applied to a vehicle device, before performing quality evaluation on the fused features through a quality evaluation model to obtain a quality score for the target video, the method also includes: quantizing and scaling the model parameters of the quality evaluation model to reduce the size of the quality evaluation model; and deploying the quantized and scaled quality evaluation model on the vehicle device.

[0025] Based on the above means, when this method is applied to vehicle equipment, due to the low hardware computing power of the vehicle, the model parameters of the quality evaluation model can be quantified and scaled to reduce the size of the quality evaluation model, thereby reducing the requirements for vehicle-side computing power and facilitating vehicle-side deployment. It can then be applied to vehicles with various computing powers and has a wide range of applications.

[0026] In a second aspect, the present application provides a video quality evaluation device, the device comprising:

[0027] An acquisition unit, used for acquiring operation features in a target video and the complexity of the target video;

[0028] A fusion unit, used to fuse the complexity and operation features to obtain fusion features of the target video;

[0029] The evaluation unit is used to evaluate the quality of the fusion features through a quality evaluation model to obtain a quality score of the target video; the quality score is used to characterize the blur degree of the target video.

[0030] In a third aspect, the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program or instructions, and when the computer program or instructions are executed by the processor, the method provided in the first aspect is implemented.

[0031] In a fourth aspect, the present application provides a vehicle device, the vehicle device includes a memory and a processor, the memory stores a computer program or instructions, and when the computer program or instructions are executed by the processor, the method provided in the first aspect is implemented.

[0032] In a fifth aspect, the present application further provides a storage medium storing a computer program or instructions, which, when executed by a processor, implements the method provided in the first aspect.

[0033] In a sixth aspect, the present application further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the method provided in the first aspect above is implemented.

[0034] It should be noted that the technical effects of the second to sixth aspects can refer to the detailed description of the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A first optional flow chart of a method for evaluating video quality provided in an embodiment of the present application;

[0036] Figure 2 A second optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0037] Figure 3 A third optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0038] Figure 4 A fourth optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0039] Figure 5 A fifth optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0040] Figure 6 A sixth optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0041] Figure 7 A seventh optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0042] Figure 8 An eighth optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0043] Fig. 9 A ninth optional flow chart of the video quality evaluation method provided in the embodiment of the present application;

[0044] Fig.10 An optional flowchart of the cooperation of various modules provided in the embodiment of the present application;

[0045] Fig.11 An optional structural diagram of a feature extraction module provided in an embodiment of the present application;

[0046] Fig.12 An optional structural diagram of a feature fusion module provided in an embodiment of the present application;

[0047] Fig.13 An optional flow structure intention of the quality evaluation module provided in the embodiment of the present application;

[0048] Fig.14 A schematic diagram of an optional structure of a video quality assessment device provided in an embodiment of the present application;

[0049] Fig.15 An optional structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0051] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] In the following description, the terms "first\second\third" are used only as examples to distinguish different objects, and do not represent a specific order or have no limitation on the order of precedence. It is understandable that "first\second\third" can be interchanged with a specific order or order of precedence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] The embodiments of the present application provide a video quality evaluation method, apparatus, device, storage medium and program product, which are applied to a multi-core MCU system.

[0055] The following describes various embodiments of the video quality evaluation method, apparatus, device, storage medium and program product provided in the embodiments of the present application. The video quality evaluation method can be implemented by a video quality evaluation device, which can be deployed in an electronic device including a memory and a processor.

[0056] In a possible implementation, the electronic device may be deployed in a vehicle device, a mobile phone, a computer, or other device with relevant data processing capabilities.

[0057] In a first aspect, the present application provides a method for evaluating video quality. Figure 1 The content shown takes the electronic device as the execution subject as an example. The process may include but is not limited to the following S101 to S103.

[0058] S101. The electronic device obtains operation features in a target video and the complexity of the target video.

[0059] The target video may be any video that needs to be evaluated for video quality. The target video here may be a video acquired in real time, or a video stored after acquisition.

[0060] The implementation of this application does not limit the data size and frame number of the target video, and can be configured according to actual needs. Generally, the target video may include N frame images, where N is greater than or equal to 2.

[0061] Motion features are used to characterize the features of each object in the target video, and may also be called change features or video features.

[0062] The complexity of the target video is used to characterize the complexity of the target video. The complexity here may include the complexity of various dimensions. For example, it may include the complexity of the time dimension and / or the complexity of the space dimension.

[0063] S102: The electronic device fuses the complexity and the operation characteristics to obtain fusion characteristics of the target video.

[0064] The electronic device adjusts the complexity to the same feature as the running feature dimension, then concatenates the two, reduces the number of channels through a 1×1 convolution layer, and finally performs activation processing through an activation function to obtain the fused features of the target video.

[0065] S103: The electronic device performs quality evaluation on the fusion features through a quality evaluation model to obtain a quality score of the target video.

[0066] The quality score is used to characterize the blurriness of the target video.

[0067] In a possible implementation manner, a higher quality score corresponds to a clearer target video; a lower quality score corresponds to a blurrier target video.

[0068] S103 can be implemented as follows: the electronic device inputs the fused features into the quality evaluation model. Since the quality evaluation model is a pre-trained model for determining the video quality score, the fused features are processed by the quality evaluation model to obtain the quality score of the target video.

[0069] The embodiment of the present application does not limit the type of quality evaluation model, and can be configured according to actual needs. For example, the quality evaluation model can be a convolutional neural network model or a binary classification model, etc.

[0070] The video quality evaluation method provided in this implementation of the present application includes obtaining the operating features in the target video and the complexity of the target video; fusing the complexity and the operating features to obtain the fused features of the target video; performing quality evaluation on the fused features through a quality evaluation model to obtain a quality score of the target video; the quality score is used to characterize the degree of blur of the target video.

[0071] In this embodiment, based on the above means, when scoring the quality of the target video, the complexity and motion features of the target video are fused, and the quality of the target video is scored based on the fused features. In practice, different videos may have different corresponding complexities. This solution takes into account the impact of the complexity of the target video on the quality score, thereby improving the accuracy of the quality score.

[0072] Next, the process of the electronic device acquiring the operation features in the target video in S101 is described.

[0073] refer to Figure 2 As shown in the content, the process may include but is not limited to the following S201 to S203.

[0074] S201. The electronic device inputs a target video into a three-dimensional convolution layer, processes the target video through the three-dimensional convolution layer, and obtains a first output.

[0075] The 3D convolutional layer is used for temporal feature extraction.

[0076] The first output is used to characterize the temporal features of the target video.

[0077] The target video's N consecutive frames are input into the 3D convolution layer, and the correlation features between the previous and next frames in the N consecutive frames are extracted through the 3D convolution layer to obtain the first output. Since the N frames are consecutive N frames, that is, they are ordered in time, the first output can represent the time features of the target video.

[0078] S202: The electronic device inputs the first output to a two-dimensional convolutional layer, processes the first output through the two-dimensional convolutional layer, and obtains a second output.

[0079] The 2D convolutional layer is used to extract spatial features.

[0080] The second output is used to characterize the temporal and spatial characteristics of the target video.

[0081] The first output is input to the two-dimensional convolution layer, and the two-dimensional convolution layer is used to extract features of each image in the first output to obtain the second output. The two-dimensional convolution layer mainly extracts the feature relationship between each pixel on each image, so the second output can represent the spatial features of the target video. The temporal features of the first output are superimposed, and the second output obtained can represent the temporal and spatial features of the target video, that is, the spatiotemporal features.

[0082] S203: The electronic device determines a motion feature based on the second output.

[0083] In a possible implementation manner, the electronic device directly determines the second output as the operating characteristic.

[0084] In another possible implementation, the electronic device further processes the second output to determine a motion characteristic.

[0085] It should be noted that when obtaining the second output, the spatial features can be first extracted through a two-dimensional convolution layer, and then input into a three-dimensional convolution layer to extract the temporal features to obtain the second output.

[0086] In practice, the order of the two-dimensional convolution layer and the three-dimensional convolution layer can be configured according to demand, and the embodiments of the present application do not make a sole limitation to this.

[0087] In this embodiment, based on the above means, when determining the motion features, the time features are first extracted through the three-dimensional convolution layer, and then the spatial features are extracted based on the two-dimensional convolution layer, so that the second output obtained can characterize the spatiotemporal features (time features and spatial features) of the target video. In this way, on the one hand, the spatiotemporal features are used as motion features, which not only consider the video quality in the time dimension, but also consider the video quality in the spatial dimension, so the information of the determined motion features is more comprehensive, so that the quality score of the determined target video is more accurate; on the other hand, the time features and spatial features are extracted respectively through the three-dimensional convolution layer and the two-dimensional convolution layer. Since the three-dimensional convolution layer and the two-dimensional convolution layer do not require high computing power, the implementation process has low hardware requirements for the device and is widely applicable.

[0088] In one possible implementation, reference Figure 3 As shown in the content, the electronic device determining the motion characteristics based on the second output in S203 may include but is not limited to the following S2031 to S2033.

[0089] S2031. The electronic device reduces the size of the second output to obtain a first size feature.

[0090] The embodiment of the present application does not limit the reduction ratio, and can be configured according to actual needs. For example, it can be 1 / 4, 1 / 8, 1 / 16, 1 / 32, etc.

[0091] The first size feature here refers to the feature after the first output is reduced in size ratio.

[0092] S2032. The electronic device superimposes the first size feature and the second output to obtain a first intermediate feature.

[0093] The electronic device obtains a first intermediate feature by splicing and superimposing the first size feature and the second output.

[0094] S2033: The electronic device determines the first intermediate feature as a motion feature, or continues to reduce the size based on the first intermediate feature to obtain a motion feature.

[0095] In a possible implementation, the electronic device determines the first intermediate feature as a motion feature.

[0096] In another possible implementation, the electronic device may reduce the size of the first intermediate feature as a new second output to obtain a second size feature, and superimpose the second size feature with the first intermediate feature to obtain a second intermediate feature. The electronic device determines the second intermediate feature as the operation feature.

[0097] Alternatively, the electronic device may continue to reduce the size of the second intermediate feature until a preset number of cycles is reached to obtain the i-th intermediate feature, and determine the i-th intermediate feature as the operating feature.

[0098] In this embodiment, based on the above means, when determining the motion feature, the second output can be further processed. During the processing, the image of the second output can be reduced in size, and then the reduced size feature and the second output are superimposed to obtain the first intermediate feature, and the motion feature is determined based on the first intermediate feature. Since the first intermediate feature is the superposition of the reduced size feature and the second output, the reduced feature and the feature information of the second output can be characterized, so that the motion feature can be highlighted in the overall (second output) and details (reduction); if the first intermediate feature is continuously reduced in size, and the cycle is repeated several times, detail features of different size dimensions and overall features can be obtained; so that the motion feature can characterize the overall feature and detail features of various size dimensions, so that the quality score determined in this way takes into account the overall quality and the detail quality of various sizes, and the obtained quality score is more accurate.

[0099] Next, the process of the electronic device acquiring the complexity of the target video in S101 is described.

[0100] The process may include but is not limited to the following method 1 and / or method 2.

[0101] Method 1: Determine the spatial complexity of the target video;

[0102] Method 2: Determine the time complexity of the target video.

[0103] refer to Figure 4 As shown in the content, the process of determining the spatial complexity of the target video in method 1 may include but is not limited to display S401 to S403.

[0104] S401: The electronic device determines a standard deviation of each frame of an image in a target video.

[0105] The electronic device can use a Sobel filter for filtering. Then the standard deviation of the pixels in each Sobel filtered frame is calculated to obtain the standard deviation of each image.

[0106] When determining the standard deviation of each image, the color value of each image is determined first, and the standard deviation of each image is determined based on the color value of each pixel in each image.

[0107] S402: The electronic device traverses all images in the target video to obtain standard deviations of the multiple images.

[0108] The electronic device repeats this operation for each frame in the target video sequence and generates a time series of scene spatial information.

[0109] S403: The electronic device determines the maximum value of the standard deviations of the multiple images as the spatial complexity of the target video.

[0110] The electronic device selects the maximum value in the time series to represent the complexity of the spatial information content of the scene, that is, determines the maximum value of the standard deviation of multiple images as the spatial complexity of the target video.

[0111] refer to Figure 5 As shown in the content, the process of determining the time complexity of the target video in Mode 2 may include but is not limited to display S501 and S502.

[0112] S501: The electronic device determines, based on the target video, a correlation standard deviation between each frame image and a previous frame image at corresponding pixels, and obtains N correlation standard deviations.

[0113] The electronic device calculates the color difference between each frame and the pixel at the same position in the previous frame to obtain a set of related standard deviations between the previous and next frames, and traverses all the previous and next frames to obtain N related standard deviations. The N related standard deviations can also be combined into a time series.

[0114] S502: The electronic device determines the maximum value of N associated standard deviations as the time complexity of the target video.

[0115] The electronic device selects the maximum value of the time series (the maximum value of N associated standard deviations) to represent the temporal information content of the scene, that is, the temporal complexity of the target video.

[0116] In this embodiment, based on the above means, the spatial complexity and / or temporal complexity can be determined based on the standard deviation. On the one hand, the complexity of the target video can be reflected in different dimensions, and the complexity can be determined more accurately. For example, some video images are relatively complex, and the corresponding spatial complexity is relatively high. Some video images are simple, but the images change more, and the corresponding temporal complexity is relatively high. In the application, the complexity can be determined based on actual needs. It can be seen that the complexity determined by this method is more accurate. On the other hand, determining the complexity by the standard deviation method makes the implementation logic simple and reliable.

[0117] refer to Figure 6 The video quality evaluation method provided in the embodiment of the present application may also include but is not limited to the following S104 and S105.

[0118] S104: When the quality score is less than a score threshold, the electronic device determines that the target video is blurred.

[0119] The embodiment of the present application does not limit the specific value of the quality score threshold, and can be configured according to actual needs. For example, the quality score threshold can be 5 points or 7 points, etc.

[0120] S105. The electronic device outputs the blurring reason of the target video through the quality evaluation model; the blurring reason includes: physical factor blurring and algorithm factor blurring.

[0121] Physical blur refers to blur caused by camera lens being dirty or blocked.

[0122] Algorithmic blur refers to video frame loss, blur, block effects, etc. caused by the algorithm.

[0123] The quality assessment model can be pre-trained to output not only the quality score but also the ambiguity reason.

[0124] In this embodiment, based on the above means, the reasons for the video module (physical factor blur and algorithmic factor blur) can also be output through the quality evaluation model. In this way, the user can know the reason for the blur of the target video in time, and the user experience is high.

[0125] Next, the process of the electronic device determining that the target video is blurred when the quality score is less than the score threshold in S104 is described.

[0126] refer to Figure 7 As shown in the content, the process may include but is not limited to the following S1041 and S1042.

[0127] S1041: When the quality score is less than a first score threshold, the electronic device determines that the target video is blurred.

[0128] Among them, the ambiguity is caused by the ambiguity of algorithmic factors.

[0129] For example, the quality scoring model is preconfigured to have a score range of 1-20, where 1-10 represents blurring due to algorithmic factors and 11-20 represents blurring due to physical factors.

[0130] Assuming that the first scoring threshold is 7 points, the electronic device determines that the quality score of the target video is less than 7 points, then the target video is blurred, and the cause of the blur is the blur caused by algorithm factors.

[0131] S1042: When the quality score is less than the second score threshold and greater than the second score starting value, the electronic device determines that the target video is blurred.

[0132] Among them, the ambiguity is caused by physical factors.

[0133] The first scoring threshold is smaller than the second scoring threshold, and the second scoring starting value is larger than the first scoring threshold.

[0134] The second scoring threshold is 17 points, and the second scoring starting value is 11.

[0135] Assuming that the second scoring threshold is 17 points, the electronic device determines that the quality score of the target video is less than 17 points and greater than 10 points, then the target video is blurred, and the cause of the blur is blur caused by physical factors.

[0136] In this embodiment, based on the above means, since there are two fuzzy factors, the corresponding scores have two score ranges. In the first score range, if the quality score is less than the first score threshold, it is determined that the target video is fuzzy due to algorithmic factors; in the second score range, if the quality score is less than the second score threshold, it is determined that the target video is fuzzy due to physical factors. The starting value of the second score range is the second score starting value. In this way, the fuzzy factor of the target video can be determined from the two fuzzy factors through a single score through a quality score model, which is simple, convenient and reliable.

[0137] The video quality assessment method provided in the embodiment of the present application may also include a reminder alarm process.

[0138] When the method is applied to vehicle equipment, the process may include: when the quality score is less than a score threshold, outputting a first reminder message; the first reminder message is used to remind that the current video is blurred to cancel the first function of the vehicle; the first function includes: intelligent driving function, assisted driving function or intelligent reversing function.

[0139] The first function may also include other functions that need to refer to the target video to be implemented, which are not listed here one by one.

[0140] The first function may be canceled here by manually canceling the first function based on the first reminder information, or the first function may be automatically canceled after the device detects that the video is blurred.

[0141] The first function that is cancelled is generally some functions in the process of intelligent driving or assisted driving. Canceling the first function can also be understood as switching to manual implementation, that is, driving, reversing, etc. through manual operation.

[0142] In this implementation, based on the above means, a reminder message can also be output when the quality score is low. In this way, the user can promptly perceive which functions may be affected by the blurry video, thereby canceling the corresponding functions and improving the safety of the vehicle. For example, when the video is blurry, the smart driving function can be canceled in time to improve driving safety.

[0143] Next, the process of obtaining the quality evaluation model provided in the application embodiment is described.

[0144] refer to Figure 8 As shown in the content, the process may include but is not limited to the following S801 and S802.

[0145] S801. The electronic device obtains training data.

[0146] The training data includes: multiple training videos, and a subjective quality score of each of the multiple training videos; a type label of each of the multiple training videos, and the type labels include clarity, blur caused by physical factors, and blur caused by algorithmic factors.

[0147] The implementation of this application does not limit the amount of training data, and can be configured according to actual needs. The larger the amount, the higher the accuracy of the trained model.

[0148] S802. The electronic device trains the convolutional neural network using the training data to obtain a quality evaluation model.

[0149] There is no limitation on the specific training process here, and it can be configured according to actual needs.

[0150] For example, configure the loss function, determine the loss value based on the output and the loss function, reversely adjust the parameters in the model based on the loss value, and repeat this cycle until the training data is trained or the model converges, and determine the converged or trained model as the quality evaluation model.

[0151] If the input training data only includes the subjective quality score of each training video in multiple training videos, the obtained quality model can output the video score; if the input training data only includes the subjective quality score of each training video in multiple training videos and the type label of each training video in multiple training videos, the type label includes clarity, blur caused by physical factors, and blur caused by algorithmic factors. The model obtained in this way can not only output the video score, but also the blur type.

[0152] Based on the above means, the convolutional neural network model can be trained with multiple training data and the subjective quality score of each training video to obtain a quality evaluation model. The quality evaluation model obtained in this way can be used to score videos. Furthermore, the convolutional neural network model can be trained with multiple training data and the subjective quality score of each training video, as well as the type label of each video, to obtain a quality evaluation model. The quality evaluation model obtained in this way can be used to score videos and can also output blur reasons.

[0153] When the method is applied to vehicle equipment, the embodiment of the present application may also include deploying the quality evaluation model on the vehicle equipment.

[0154] refer to Fig. 9 As shown in the content, the process may include but is not limited to the following S901 and S902.

[0155] S901: The electronic device quantizes and scales the model parameters of the quality evaluation model to reduce the size of the quality evaluation model.

[0156] For example, in order to improve the model inference speed and obtain a smaller model size and lower memory usage, the electronic device can use int8 quantization for the output quality evaluation model and map the float32 data to a low-bit value, thereby reducing the size of the quality evaluation model.

[0157] Generally speaking, a quantized model, that is, a parameter in a quantized model, for example, maps the parameter in the model from float32 data to a low-bit value.

[0158] S902: The electronic device deploys the quantized and scaled quality evaluation model on the vehicle device.

[0159] Based on the above means, when this method is applied to vehicle equipment, due to the low hardware computing power of the vehicle, the model parameters of the quality evaluation model can be quantified and scaled to reduce the size of the quality evaluation model, thereby reducing the requirements for vehicle-side computing power and facilitating vehicle-side deployment. It can then be applied to vehicles with various computing powers and has a wide range of applications.

[0160] Below, the video content captured by the vehicle-mounted camera is taken as an example to illustrate the video quality evaluation process provided by the embodiment of the present application.

[0161] During the video transmission process, due to the influence of data transmission and various video processing algorithms, the video quality will be attenuated compared to the original video, resulting in potential problems in subsequent use, and also having a certain impact on people's visual experience. In modern cars, with the promotion of technologies such as advanced driver assistance, the video collected by the vehicle camera will also be transmitted to downstream recognition and detection tasks. If the collected video is blurred, it will have a huge impact on subsequent tasks. Therefore, how to achieve real-time monitoring of video quality at a faster speed and lower cost to avoid potential risks has become a new research challenge.

[0162] From the perspective of method and technical implementation, there are mainly the following difficulties:

[0163] 1. In practice, evaluation needs to be performed without the original reference video.

[0164] Currently, most studies focus on quality assessment under the condition of clear reference video. For example, in Solution 1, the difference between the tested video sequence and the reference video sequence is used to obtain the objective spatial domain distortion. The objective spatial domain distortion is processed by using the spatial domain visual characteristics of the human eye to obtain the subjective spatial domain distortion. The video quality is evaluated based on the subjective time domain changes, and it has a high correlation with the evaluation results of the subjective video quality assessment method. However, in actual applications, there is often no clear video stream for comparison, so Solution 1 has certain limitations.

[0165] 2. Serial insertion processing components require high processing time.

[0166] In Solution 2, a no-reference video quality assessment method is disclosed. Through a multi-branch recurrent neural network model, the quality-related time domain distortion is effectively learned. In the face of videos with complex motion scenes, the quality can be accurately predicted, a universal no-reference video quality evaluation model is established, and no-reference quality evaluation of distorted videos is automatically realized.

[0167] However, when applied in vehicle-side scenarios, it is often necessary to insert data into the processing process in parallel. If a recurrent neural network is used, a large amount of calculation will be introduced, and there is a risk of low computing efficiency.

[0168] Therefore, considering the above problems, this embodiment of the present application designs a video quality monitoring system, which uses a simple convolutional neural network to achieve real-time monitoring of video quality, and can automatically determine whether the source type of blur is due to physical factors or algorithmic factors, and at the same time give a score for the degree of blur. When the score indicates that the video quality is poor, the driver will be reminded to pay attention to the surrounding environment.

[0169] The purpose of this embodiment is to provide a video quality detection method for the vehicle video system, which can complete real-time video quality detection, thereby improving the reliability and safety of driving. Its main contents include: data set organization, core algorithm construction, model training, and vehicle-side deployment.

[0170] 1. Data set organization.

[0171] Based on the current open source dataset for video quality assessment, the sources of blur are classified into two categories: physical factors and algorithmic factors. Physical factors include camera lens contamination and occlusion, while algorithmic factors include video frame loss, blur, block effects, etc. caused by the algorithm. The dataset contains continuous frames of video with impaired quality and the corresponding subjective evaluation scores of the video. At the same time, in order to facilitate the processing of subsequent algorithms, the image frame data in YUV format needs to be converted into RGB data.

[0172] 2. Core algorithm construction.

[0173] After the processed video frames are obtained, the video quality evaluation will be realized through the constructed neural network. The algorithm network is mainly divided into the following modules:

[0174] 2.1 Feature Extraction Module

[0175] This module can process the input continuous frames through convolutional neural networks and extract motion features of objects such as vehicles and pedestrians.

[0176] 2.2 Feature Fusion Module

[0177] The function of this module is to calculate the time complexity and space complexity information of several input frames, fuse and reconstruct them with the features obtained by the neural network, and obtain the fused features.

[0178] 2.3 Quality Evaluation Module

[0179] This module is divided into two steps. The first step is to determine whether the blur in the video is caused by physical factors or algorithmic factors. The second step is to calculate the blur degree according to the blur type and get a score for the video quality.

[0180] 3. Model training

[0181] After completing data processing and algorithm construction, it needs to be trained. Training is divided into two parts:

[0182] 3.1. Blur type judgment. In this step of training, the model finally needs to judge whether the blur in the video is caused by physical factors or algorithmic factors, and pre-train the network at the same time;

[0183] 3.2. Blur degree judgment. In this step of training, the model needs to use two sets of data sets to train the blur degrees of the two blur types respectively, match the training with the subjective evaluation scores annotated in the data sets, and use the scores obtained by the network to fit the subjective scores obtained manually.

[0184] 4. Vehicle-side deployment

[0185] The CPU and GPU currently used in car computers require high computing power when processing algorithms such as Transformer, and some domestic chips do not even support it. Therefore, this application mainly uses convolutional networks to implement the reasoning process, reduce the computational difficulty of the chip, and realize convenient deployment of the model.

[0186] refer to Fig.10 As shown in the content, the process of cooperation of each module may include but is not limited to: inputting the input video 1001 to the feature extraction module 1002, extracting features through the feature extraction module 1002 and inputting the features to the feature fusion module 1003, and after the feature fusion module 1003 performs feature fusion, the fused features are input to the quality evaluation module 1004. The quality evaluation module 1004 performs fuzzy category judgment 1005 on the fused features, and obtains the quality evaluation score 1006 corresponding to the physical factor and the quality evaluation score 1007 corresponding to the algorithm factor.

[0187] The implementation principle of this embodiment includes: a video quality monitoring system is designed through four aspects: data set organization, core algorithm construction, model training and vehicle-side deployment. When the vehicle collects video, the video source will be processed on the chip to obtain the evaluation score of the video collected in the current time period. When it is lower than the threshold, it indicates that there is a quality problem with the collected video, and the driver will be reminded and informed of the cause of the problem, whether it is due to physical factors or internal algorithm factors. It is conducive to the troubleshooting of problems and driving safety.

[0188] There are two main application scenarios:

[0189] Improve the efficiency of vehicle data collection. When collecting vehicle video data, this method can be used to determine in real time whether the quality of the collected video meets the standard, which is sufficient for training other algorithms. Traditional methods often require manual screening by labelers, which consumes a lot of time and manpower costs. This method can greatly improve efficiency.

[0190] Driving safety warning. When the vehicle is using assisted driving related functions or watching reversing images, if the video is blurry, there will be potential safety hazards. When this method detects that the video quality score is lower than the threshold, it can remind the driver to improve driving safety.

[0191] This embodiment is a video quality monitoring system with a video quality evaluation algorithm as the core, which can effectively improve the efficiency of vehicle data collection and improve driving safety.

[0192] Including but not limited to: A feature extraction and fusion technology for video data, which extracts features through a convolutional network and then fuses them with the time information contained in the video, making full use of the correlation between adjacent frames of the video for feature extraction. A blur type judgment model. Using the trained feature extraction module, it is possible to analyze multiple consecutive frames in the video, and then determine the cause of the video blur, which is conducive to troubleshooting. A reference-free quality evaluation method. Using the trained model, the quality loss in time and space is analyzed at the same time, and video quality evaluation is achieved without reference by analyzing the spatiotemporal features.

[0193] This embodiment can improve the collection efficiency of the vehicle's video data collection through the quality evaluation algorithm, quickly exclude low-quality data, and reduce the personnel input and cost of subsequent data cleaning. In addition, data of different qualities can be preliminarily classified to provide data for the training of algorithms such as rain removal and defogging in video processing. Using this algorithm, the vehicle's video collection system can be evaluated to avoid safety hazards caused by external camera contamination or algorithm problems, thereby improving driving safety.

[0194] Next, the implementation process of video quality evaluation is described.

[0195] Step 1: Dataset annotation.

[0196] We perform secondary annotation on datasets such as video extensions to distinguish blur types, including blur caused by physical factors such as raindrops, dirt, or other stains, and blur caused by algorithmic factors such as frame loss, aliasing, or block effects in the video. In addition, we normalize the subjective evaluation scores of the video evaluations in the dataset and scale the score range to 1-10.

[0197] The normalization operation can refer to the description of the following formula (1).

[0198]

[0199] In formula (1), s represents the subjective quality evaluation score of the dataset itself, s ′ It represents the normalized quality evaluation score, min(s) represents the lowest score, and max(s) represents the highest score.

[0200] Step 2: Preprocess the data.

[0201] For some video data, each video consists of several frames of raw image data in YUV format, which needs to be converted into RGB data. The conversion process can refer to the description of formula (2-1) to formula (2-3).

[0202] R = 1.164 × (Y-16) + 2.018 × (V-128) Formula (2-1);

[0203] G=1.164×(Y-16)-0.813×(U-128)-0.391×(V-128) Formula (2-2);

[0204] B=1.164×(Y-16)+1.596×(U-128) Formula (2-3);

[0205] In formula (2-1) to formula (2-3), YUV represents the color value in the YUV format before conversion in the image, and RGB represents the color value in the YUV format after conversion in the image.

[0206] Step 3: Build a video quality evaluation algorithm model.

[0207] This embodiment consists of three modules: feature extraction, feature fusion and quality evaluation.

[0208] The establishment of different modules will be specifically introduced below mainly in conjunction with the accompanying drawings.

[0209] Step 2.1: Building the feature extraction module.

[0210] Fig.11 Schematic diagram of the feature extraction module. Fig.11 As shown in the content, the input is n frames of continuous images 1101. The n frames of continuous images 1101 are first input into the 3D convolution layer 1102 for temporal feature extraction, and then input into the 2D convolution layer 1103 for spatial feature extraction. After that, they are divided into four branches and respectively input into the 1 / 4 feature branch 1104, the 1 / 8 feature branch 1105, the 1 / 16 feature branch 1106, and the 1 / 32 feature branch 1107, and finally the multi-scale feature 1108 is obtained.

[0211] The input of the model is video, that is, n consecutive frames, and the images x1~x nThe deep features are upsampled in turn and added to the previous scale features to obtain multi-scale feature maps. The calculation method for each step can refer to formula (3).

[0212] w i ′=w i +upsample(w i+1 ) formula (3);

[0213] In formula (3), w i represents the feature of the previous scale, w i ′The features of the current scale, upsample(w i+1 ) represents the sampling of the current scale.

[0214] Step 2.2: Build the feature fusion module.

[0215] Fig.12 Schematic diagram of the feature fusion module. Fig.12 As shown in the content, the feature fusion process may include: inputting n frames of continuous images 1201 to the spatial complexity branch 1202 and the temporal complexity branch 1203 respectively. The spatial complexity branch 1202 determines the spatial complexity 12021 to obtain the temporal complexity array 12022; the temporal complexity branch 1203 determines the temporal complexity 12031 to obtain the temporal complexity array 12032; after performing a splicing operation 1205 on the temporal complexity array 12022, the temporal complexity array 12032 and the multi-scale features 1204, a spliced ​​feature 1206 is obtained.

[0216] In this module, the spatial complexity (SI) and temporal complexity (TI) of the current continuous frame are first calculated.

[0217] Spatial complexity SI usually represents a measure of the amount of spatial detail in an image. As described in formula (4), first use a Sobel filter to filter. Then calculate the standard deviation of the pixels in each Sobel filtered frame. Repeat this operation for each frame in the video sequence and generate a time series of scene spatial information. Select the maximum value in the time series to represent the spatial information content of the scene.

[0218] SI = max{std[Sobel(F n )]} Formula (4);

[0219] In formula (4), SI represents space complexity, and SI represents std[Sobel(F n ) represents the standard deviation of pixels in each Sobel filter frame, and max represents the maximum value.

[0220] Temporal complexity TI represents the temporal variation of a video sequence. Sequences with higher degrees of motion usually have higher TI values. Referring to the contents of formula (5-1) and formula (5-2), the difference between the pixels at the same position of each frame and the previous frame is first calculated, and then the standard deviation of the current frame is calculated. Similar to the calculation of SI, the maximum value of the time series is selected to represent the temporal information content of the scene.

[0221] TI=max{std[M n (i,j)]} Formula (5-1);

[0222] M n (i,j)= F n (i,j)–F n-1 (i,j) Formula (5-2);

[0223] In formula (5-1) and formula (5-2), TI represents the space complexity, F n (i, j) is the pixel value of the nth frame image at (i, j), std represents the standard deviation, and max represents the maximum value.

[0224] After the calculation is completed, refer to the content shown in formula (6) to convert SI and TI into an array with the same channel number as the feature map, and then concatenate them with the features obtained in the previous step, and then reduce the number of channels through a 1×1 convolution layer. Finally, the output features (equivalent to the above fusion features) are obtained through the relu activation function.

[0225] y out =relu(conv(concat(y,y SI ,y TI ))) Formula (6);

[0226] In formula (6), y out Represents the output feature; y is the feature obtained in the previous step, y SI ,y TI It is a one-dimensional array obtained by expanding SI and TI; relu represents the activation function, conv represents the convolution operation, and concat represents the concatenation operation.

[0227] Step 2.3: Construction of quality evaluation module.

[0228] Fig.13 It is a schematic diagram of the quality evaluation module.

[0229] refer to Fig.13 As shown, the output feature 1301 is input to the fully connected layer (FC) 1302, and then input to the activation layer 1303 after being processed by the FC layer 1302, and then the evaluation score 1304 is obtained after being processed by the activation layer 1303. The evaluation score may include: a physical factor fuzzy evaluation score 13041 and / or an algorithm factor module evaluation score 13042.

[0230] In this step, the corresponding fuzzy type and evaluation score are obtained through the fully connected layer and the activation layer. The method for calculating the fuzzy type can refer to formula (6).

[0231] out = Act_layer(FC_layer(y out )) Formula (6);

[0232] In formula (6), out represents the output score, y out Represents the obtained fusion features, Act_layer represents the activation function of the activation layer, and FC_layer represents the processing of the fully connected layer.

[0233] When performing coarse classification, Act_layer selects the sigmod activation function and rounds the output to 0 and 1. 0 indicates blur caused by algorithmic factors, and 1 indicates blur caused by physical factors. When performing the second stage of detailed classification training, the softmax activation function is selected. To facilitate score judgment, the original range of the output out is a floating point number of 0-1, which is rounded / proportionally expanded. For example, when performing detailed classification, it can be proportionally expanded to 0-20. 0-10 indicates blur caused by the algorithm, and its corresponding evaluation score is 0-10. 10-20 indicates blur caused by physical factors, and the corresponding evaluation score score' needs to be calculated as follows: score ′ =score-10.

[0234] Step 4: Network training.

[0235] In this embodiment of the present application, the network is trained using hierarchical classification in two stages.

[0236] Step 3.1: In the first stage, 40% of the training set data is used for training, and the network only outputs 0 or 1, 0 indicates blurring due to algorithmic factors, and 1 indicates blurring due to physical factors. Therefore, it can be regarded as a classification task, and the cross entropy loss function is used as the loss function.

[0237] Step 3.2: In the second stage, the sigmoid activation function is replaced with the softmax activation function, and the network is retrained with the remaining 60% of the training set data for fine classification, and the scores obtained by the network are fitted with the subjective scores obtained manually. The cross entropy loss function is also selected as the loss function.

[0238] The loss function calculation can refer to formula (7).

[0239] Loss=-∑(y i ×log(P i )) Formula (7);

[0240] In formula (7), y i Indicates whether the score is the same as the true label when it is i, if it is the same, it is 1, otherwise it is 0, P i Represents the probability of the original output of the activation layer when the score is i.

[0241] Step 5: Lightweight deployment of vehicle computers.

[0242] This invention will be applied to the evaluation of video data collected by vehicle-mounted cameras and will be deployed on vehicle-mounted chips. To achieve the lowest possible latency, the following lightweight deployment measures will be taken:

[0243] Step 4.1: The network structure adopts convolutional neural network.

[0244] The mainstream chips on current car computers can run the network with full convolution structure very well and can achieve fast reasoning. Therefore, in this embodiment of the present application, the network structure is composed of full convolution to reduce the computational difficulty of the chip and facilitate the deployment and fast reasoning of the model.

[0245] Step 4.2: Int8 quantization

[0246] In order to improve the model reasoning speed, obtain a smaller model size and lower memory usage, this application adopts int8 quantization for the output model and maps float32 data to low-bit values.

[0247] For details, please refer to formula (8).

[0248]

[0249] In formula (8), s represents the scaling factor, and Z represents the zero point coordinate of the mapping, wherein the calculation method of formula (8) can be expressed as the following formula (9).

[0250]

[0251] In formula (9), when X is less than a, the value is a; when X is greater than c, the value is c; when X is between a and c, the value is x; a can be -2 b-1 , c can be -2 b-1 , x can be

[0252] Step 6: Scenario application.

[0253] (1) Real-time data screening

[0254] When collecting video data from the vehicle, when obtaining the quality evaluation score obtained in step 2, if the score is not less than 7 points, the collected video is considered to be relatively clear; if the score is less than 7 points but not less than 4 points, it indicates that the current video is somewhat blurred and personnel are required to judge whether it can be used as valid data; when the score is less than 4 points, it is considered that the video is seriously damaged. When the original score is not higher than 10 points, the tester is prompted to check the video processing algorithm, otherwise the physical environment such as the camera is checked.

[0255] (2) Driving safety warning. After completing the deployment of step 4, when the vehicle is started, the video quality will be checked regularly. Similar to the real-time data screening scenario, if the score is not less than 7 points, the vehicle will not remind the driver; if the score is less than 7 points but not less than 4 points, the driver will be reminded to pay attention to safety while driving, and it is recommended to troubleshoot the problem; when the score is less than 4 points, the system will issue a warning to remind the driver to check the problem as soon as possible to ensure driving safety.

[0256] In a second aspect, the present application embodiment provides a video quality evaluation device, referring to Fig.14 As shown, the video quality evaluation device 140 includes an acquisition unit 1401 , a fusion unit 1402 and an evaluation unit 1403 .

[0257] An acquisition unit 1401 is used to acquire the operation characteristics in the target video and the complexity of the target video;

[0258] A fusion unit, used to fuse the complexity and operation features to obtain fusion features of the target video;

[0259] The evaluation unit is used to evaluate the quality of the fusion features through a quality evaluation model to obtain a quality score of the target video; the quality score is used to characterize the blur degree of the target video.

[0260] In some embodiments, the acquisition unit 1401 is also used to: input the target video into a three-dimensional convolution layer, process the target video through the three-dimensional convolution layer to obtain a first output; the three-dimensional convolution layer is used to extract temporal features; the first output is used to characterize the temporal features of the target video; input the first output into a two-dimensional convolution layer, process the first output through the two-dimensional convolution layer to obtain a second output; the two-dimensional convolution layer is used to extract spatial features; the second output is used to characterize the temporal features and spatial features of the target video; and determine the motion features based on the second output.

[0261] In some embodiments, the acquisition unit 1401 is also used to: reduce the size of the second output to obtain a first size feature; superimpose the first size feature with the second output to obtain a first intermediate feature; determine the first intermediate feature as a motion feature, or continue to reduce the size based on the first intermediate feature to obtain a motion feature.

[0262] In some embodiments, the acquisition unit 1401 is further used to: determine the standard deviation of each frame of the target video; traverse all images in the target video to obtain the standard deviations of multiple images; determine the maximum value of the standard deviations of the multiple images as the spatial complexity of the target video;

[0263] And / or, based on the target video, determine the correlation standard deviation between each frame image and the previous frame image at corresponding pixels to obtain N correlation standard deviations; and determine the maximum value of the N correlation standard deviations as the time complexity of the target video.

[0264] In some embodiments, the video quality assessment device 140 may further include an output unit, which is used to: determine that the target video is blurred when the quality score is less than a score threshold; output the cause of the blur of the target video through a quality assessment model; the cause of the blur includes: physical factor blur and algorithmic factor blur.

[0265] In some embodiments, the output unit is also used to: when the quality score is less than a first score threshold, determine that the target video is blurred, wherein the cause of the blur is algorithmic factor blur; when the quality score is less than a second score threshold and greater than a second score starting value, determine that the target video is blurred, wherein the cause of the blur is physical factor blur; wherein the first score threshold is less than the second score threshold, and the second score starting value is greater than the first score threshold.

[0266] In some embodiments, the video quality evaluation device 140 may further include a reminder unit, which is used to: output a first reminder message when the quality score is less than a score threshold; the first reminder message is used to remind that the current video is blurred to cancel the first function of the vehicle; the first function includes: intelligent driving function, assisted driving function or intelligent reversing function.

[0267] In some embodiments, the video quality assessment device 140 may further include a training unit, which is used to: obtain training data; the training data includes: multiple training videos, and a subjective quality score of each training video in the multiple training videos; a type label for each training video in the multiple training videos, the type label includes clarity, blur caused by physical factors, and blur caused by algorithmic factors; and train the convolutional neural network through the training data to obtain a quality assessment model.

[0268] In some embodiments, the video quality assessment device 140 may further include a quantization unit, which is used to: quantize and scale the model parameters of the quality assessment model to reduce the size of the quality assessment model; and deploy the quantized and scaled quality assessment model on the vehicle equipment.

[0269] In a third aspect, an embodiment provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program or instructions, and when the computer program or instructions are executed by the processor, the method provided in the first aspect is implemented.

[0270] In one example, reference Fig.15 As shown, the electronic device 150 includes: a processor 1501, at least one communication bus 1502, a user interface 1503, at least one external communication interface 1504 and a memory 1505. The communication bus 1502 is configured to realize the connection and communication between these components. The user interface 1503 may include a device for receiving user input, and the external communication interface 1504 may include a standard wired interface and a wireless interface.

[0271] The memory 1505 is configured to store instructions and applications executable by the processor 1501, and can also cache data to be processed or processed by the processor 1501 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0272] In a fourth aspect, the present application provides a vehicle device, the vehicle device includes a memory and a processor, the memory stores a computer program or instructions, and when the computer program or instructions are executed by the processor, the method provided in the first aspect is implemented.

[0273] In a fifth aspect, an embodiment provides a storage medium, that is, a computer-readable storage medium, on which a computer program or instructions are stored. When the computer program or instructions are executed by a processor, the method provided in the first aspect is implemented.

[0274] In a sixth aspect, an embodiment provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the method provided in the first aspect is implemented.

[0275] It should be noted here that the description of the above storage medium, device, apparatus and program product embodiments is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium, device, apparatus and program product embodiments, please refer to the description of the method embodiment for understanding.

[0276] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment. The above-mentioned embodiment sequence numbers are for description only and do not represent the advantages and disadvantages of the embodiments.

[0277] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0278] In the several embodiments provided in this embodiment, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0279] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0280] In addition, all functional units in each embodiment may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0281] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0282] Alternatively, if the above-mentioned integrated unit is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment can be essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods of each embodiment. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0283] The above are only embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the disclosed technical scope, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video quality evaluation method, characterized in that: The method comprises: Obtaining running features in a target video and the complexity of the target video; Fusing the complexity and the operation feature to obtain a fusion feature of the target video; The quality of the fusion feature is evaluated through a quality evaluation model to obtain a quality score of the target video; the quality score is used to characterize the blurriness of the target video.

2. The method according to claim 1, characterized in that The obtaining of motion features in the target video includes: Inputting the target video into a three-dimensional convolution layer, processing the target video through the three-dimensional convolution layer to obtain a first output; the three-dimensional convolution layer is used to extract temporal features; the first output is used to characterize the temporal features of the target video; Inputting the first output into a two-dimensional convolutional layer, processing the first output through the two-dimensional convolutional layer to obtain a second output; the two-dimensional convolutional layer is used to extract spatial features; the second output is used to characterize the temporal features and spatial features of the target video; The motion characteristic is determined based on the second output.

3. The method according to claim 2, characterized in that The determining the operating characteristic based on the second output comprises: Reducing the size of the second output to obtain a first size feature; Superimposing the first size feature with the second output to obtain a first intermediate feature; The first intermediate feature is determined as the motion feature, or the first intermediate feature is continuously reduced in size to obtain the motion feature.

4. The method according to any one of claims 1 to 3, characterized in that: The complexity of obtaining the target video includes: For each frame of the target video, determine the standard deviation of the image; Traversing all images in the target video to obtain standard deviations of the multiple images; Determine the maximum value of the standard deviations of the multiple images as the spatial complexity of the target video; and / or, Based on the target video, determining the correlation standard deviation between each frame image and the previous frame image at corresponding pixels, and obtaining N correlation standard deviations; The maximum value of the N associated standard deviations is determined as the time complexity of the target video.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the quality score is less than a score threshold, determining that the target video is blurred; The blurring reasons of the target video are outputted through the quality assessment model; the blurring reasons include: physical factor blurring and algorithm factor blurring.

6. The method according to claim 5, characterized in that When the quality score is less than a score threshold, determining that the target video is blurred includes: When the quality score is less than a first score threshold, determining that the target video is blurred, wherein the blurring is caused by algorithmic factor blurring; When the quality score is less than the second score threshold and greater than the second score starting value, determining that the target video is blurred, wherein the blurring is caused by physical factors; The first scoring threshold is smaller than the second scoring threshold, and the second scoring starting value is larger than the first scoring threshold.

7. The method according to any one of claims 1 to 3, characterized in that: When the method is applied to a vehicle device, the method further includes: When the quality score is less than the score threshold, a first reminder message is output; the first reminder message is used to remind that the current video is blurred so as to cancel the first function of the vehicle; the first function includes: intelligent driving function, assisted driving function or intelligent reversing function.

8. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire training data; the training data includes: a plurality of training videos, and a subjective quality score of each of the plurality of training videos; a type label of each of the plurality of training videos, wherein the type label includes clarity, blur caused by physical factors, and blur caused by algorithmic factors; The convolutional neural network is trained using the training data to obtain the quality evaluation model.

9. The method according to any one of claims 1 to 3, characterized in that: When the method is applied to a vehicle device, before performing the quality evaluation of the fusion feature by the quality evaluation model to obtain the quality score of the target video, the method further includes: Quantizing and scaling the model parameters of the quality assessment model to reduce the size of the quality assessment model; The quantized and scaled quality assessment model is deployed on vehicle equipment.

10. A video quality evaluation device, characterized in that: The device comprises: An acquisition unit, used for acquiring operation features in a target video and the complexity of the target video; A fusion unit, used for fusing the complexity and the operation feature to obtain a fusion feature of the target video; An evaluation unit is used to perform quality evaluation on the fusion feature through a quality evaluation model to obtain a quality score of the target video; the quality score is used to characterize the blur degree of the target video.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program or instructions, and when the processor executes the computer program or instructions, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or an instruction, and when the processor executes the computer program or the instruction, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The computer program product includes a computer program or instructions, and when a processor executes the computer program or instructions, the method according to any one of claims 1 to 9 is implemented.

Images