Model training method, video processing method, electronic device and readable storage medium

Abstract

The present application discloses a model training method, a video processing method, an electronic device and a readable storage medium. The model training method includes: obtaining a video data set with labels in a target business scenario; extracting sample key frames from the video in the video data set to obtain a global information video frame data set; locating local information in the sample key frames and cropping them to obtain a local information video frame data set; determining the global information video frame data set and the local information video frame data set as a low-quality video data set; obtaining an initial low-quality video attribution recognition model; inputting a sample of the low-quality video data set into the initial low-quality video attribution recognition model to obtain an attribution recognition result corresponding to the sample; based on the attribution recognition result of the sample and the corresponding label, adjusting the initial low-quality video attribution recognition model to obtain a trained low-quality video attribution recognition model. The present application can obtain a low-quality video attribution recognition model with a higher attribution recognition accuracy.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to a model training method, a video processing method, an electronic device and a readable storage medium. Background Art

[0002] The video clarity varies due to factors such as recording equipment, light, and network transmission. Generally, low-quality videos show various types of overexposure, noise, masking, stray light, etc., resulting in poor video playback and affecting the viewing experience.

[0003] Some solutions for attribution identification of low-quality videos can effectively detect different types of low-quality factors and provide targeted improvement suggestions for improving video quality. Specifically, these solutions use deep learning methods to construct video datasets and perform low-quality classification annotations, extract features based on the resnet-18 network, and directly perform classification processing. However, the entire method of this solution is relatively conventional, and the attribution identification accuracy is not high due to certain limitations in feature extraction capabilities.

[0004] In summary, how to effectively solve problems such as low-quality video attribution identification is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of this application is to provide a model training method, a video processing method, an electronic device and a readable storage medium, which can effectively improve the accuracy of attribution recognition by combining the global information features and local information features in the video to perform low-quality video attribution recognition.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] A model training method, comprising:

[0008] Obtain a labeled video dataset for the target business scenario;

[0009] Extracting sample key frames from the videos in the video data set to obtain a global information video frame data set;

[0010] Locating the local information in the sample key frame and cropping it to obtain a local information video frame data set;

[0011] Determine the global information video frame dataset and the local information video frame dataset as the low-quality video dataset;

[0012] Obtain an initial low-quality video attribution recognition model;

[0013] Inputting samples of the low-quality video data set into the initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples;

[0014] Based on the attribution recognition result of the sample and the corresponding label, the initial low-quality video attribution recognition model is adjusted to obtain a trained low-quality video attribution recognition model.

[0015] Preferably, based on the attribution recognition result of the sample and the corresponding label, adjusting the initial low-quality video attribution recognition model to obtain a trained low-quality video attribution recognition model includes:

[0016] Comparing the attribution recognition result corresponding to the sample with the corresponding label to obtain a comparison result; wherein the comparison result is the corresponding low-quality video attribution recognition accuracy rate;

[0017] Calculate the model loss value using the low-quality video attribution recognition accuracy;

[0018] When the model loss value reaches a preset standard, the iterative adjustment of the initial low-quality video attribution recognition model is stopped to obtain the low-quality video attribution recognition model.

[0019] Preferably, the step of obtaining a video dataset with labels in a target business scenario includes:

[0020] Build a diverse index video dataset for live broadcast business scenarios and add low-quality live broadcast video samples to obtain a low-quality live broadcast video dataset;

[0021] Classification labels are marked for video samples in the low-quality live video data set, and video samples that do not meet preset conditions are filtered to obtain the labeled video data set.

[0022] Preferably, extracting sample key frames from the video in the video data set to obtain a global information video frame data set includes:

[0023] Extracting key frames from the videos in the video dataset;

[0024] After edge filling processing is performed on the key frame, bilinear interpolation scaling is performed on the key frame according to a preset size to obtain the global information video frame.

[0025] Preferably, local information in the sample key frame is located and cropped to obtain a local information video frame data set, including:

[0026] Detecting an abnormal point in the global information video frame, and determining the abnormal point as the local information center point;

[0027] The global information video frame is cropped using the abnormal points to obtain the local information video frame.

[0028] Preferably, inputting samples of the low-quality video data set into the initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples includes:

[0029] Using the global information video frame and the local information video frame as inputs of a two-way network of the initial low-quality video attribution recognition model;

[0030] Respectively extracting global features of the global information video frame and local features of the local information video frame;

[0031] Performing feature mapping on the global features and the local features along the channel dimension to obtain a global feature map and a local feature map with the same dimension;

[0032] Fusion of the global feature map and the local feature map to obtain a mixed bilinear feature map;

[0033] The mixed bilinear feature map is used to obtain attribution recognition results corresponding to samples in the low-quality video data set.

[0034] Preferably, fusing the global feature map and the local feature map to obtain a mixed bilinear feature map includes:

[0035] Pooling the global feature map and the local feature map to obtain a global feature pooling vector and a local feature pooling vector;

[0036] Performing point-to-point inner product processing on the global feature vector and the local feature pooling vector to obtain the mixed bilinear feature map.

[0037] Preferably, using the mixed bilinear feature map to obtain the attribution recognition result corresponding to the sample in the low-quality video data set includes:

[0038] Pooling the mixed bilinear feature map to obtain a mixed feature vector;

[0039] Connecting the mixed feature vector to a fully connected layer, and outputting the category probability distribution of each video low-quality category through a normalized exponential function;

[0040] The attribution recognition result is determined using the category probability distribution.

[0041] A video processing method, comprising:

[0042] Acquire a target video, and extract key frames from the target video to obtain a target global information video frame;

[0043] Locating local information of the target global information video frame and cropping it to obtain a target local information video frame;

[0044] Inputting the target global information video frame and the target local information video frame into the low-quality video attribution recognition model trained by the above-mentioned model training method to perform low-quality video attribution recognition, and obtaining a target attribution recognition result;

[0045] Based on the target attribution recognition result, a video quality improvement suggestion is output.

[0046] An electronic device, comprising:

[0047] Memory for storing computer programs;

[0048] A processor is used to implement the steps of the above-mentioned video processing method when executing the computer program.

[0049] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video processing method are implemented.

[0050] Apply the method provided in the embodiment of the present application to obtain a labeled video dataset in a target business scenario; extract sample key frames from the video in the video dataset to obtain a global information video frame dataset; locate local information in the sample key frames and crop them to obtain a local information video frame dataset; determine the global information video frame dataset and the local information video frame dataset as low-quality video datasets; obtain an initial low-quality video attribution recognition model; input samples of the low-quality video dataset into the initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples; based on the attribution recognition results and corresponding labels of the samples, adjust the initial low-quality video attribution recognition model to obtain a trained low-quality video attribution recognition model.

[0051] In the present application, a labeled video dataset in the target business scenario is first obtained, and then sample key frames are extracted from the videos in the video dataset to obtain a global information video frame dataset. Then, the local information of the global information video frame is located and cropped to obtain a local information video frame dataset. The global information video frame dataset and the local information video frame dataset are determined as low-quality video datasets, and samples of the low-quality video dataset are input into the initial low-quality video attribution recognition model to obtain the attribution recognition results corresponding to the samples. Based on the attribution recognition results and corresponding labels of the samples, the initial low-quality video attribution recognition model is adjusted to obtain a trained low-quality video attribution recognition model. Since the training of the low-quality video attribution recognition model not only pays attention to global features but also local features, it can effectively improve the accuracy of the attribution recognition results of the low-quality video attribution recognition model finally trained. Furthermore, the video quality improvement suggestions can be made more accurate and more conducive to improving video quality.

[0052] Correspondingly, the embodiments of the present application also provide an electronic device and a readable storage medium corresponding to the above-mentioned video processing method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 This is a flow chart for implementing a model training method in an embodiment of the present application;

[0055] Figure 2 This is a flowchart of a video processing method in an embodiment of the present application;

[0056] Figure 3 This is a schematic diagram of an implementation of low-quality video attribution identification in an embodiment of the present application;

[0057] Figure 4 A flowchart of a low-quality video attribution identification method in an embodiment of the present application;

[0058] Figure 5 This is a schematic diagram of a model training device in an embodiment of the present application;

[0059] Figure 6 is a schematic diagram of a video processing device in an embodiment of the present application;

[0060] Figure 7This is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0061] Figure 8 This is a schematic diagram of the specific structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0063] Please refer to Figure 1 , Figure 1 This is a flow chart of a model training method in an embodiment of the present application, the method comprising the following steps:

[0064] S100: Obtain a video dataset with labels in a target business scenario.

[0065] Among them, the target business scenario can correspond to the specific application scenario of the low-quality video attribution recognition model that needs to be trained, such as live broadcast scenario, broadcast scenario, etc.

[0066] In this embodiment, the video dataset is a labeled dataset, that is, the video samples in the video dataset have correct low-quality video attribution, such as the specific type of cause that causes the low quality of the target video, such as overexposure, noise, masking, stray light, normal, unknown, etc.

[0067] In a specific implementation of the present application, when the target business scenario is a live broadcast business scenario, step S100 obtains a video dataset with labels in the target business scenario, including:

[0068] Step 1: Build a diverse index video dataset for live broadcast business scenarios, and add low-quality live broadcast video samples to obtain a low-quality live broadcast video dataset;

[0069] Step 2: Label the video samples in the low-quality live video dataset with classification labels, and filter out video samples that do not meet the preset conditions to obtain a labeled video dataset.

[0070] That is, first, we can build a diversified live video data set for the live broadcast business scenario, manually screen and add low-quality live video samples in batches, and perform data distribution balancing to make the low-quality video reach an appropriate proportion (such as 0.5, of course, it can also be other appropriate ratios, which can be obtained through practical data statistics, and will not be listed here one by one).

[0071] Then, the low-quality live video dataset can be screened again to filter out video samples that do not meet the requirements (such as low quality caused by network transmission, which is difficult for the video recorder to improve), and objective classification and labeling are performed at the same time. The label types include overexposure, noise, masking, stray light, normal, unknown, etc., to form a labeled video dataset.

[0072] S200 , extracting sample key frames from the video in the video data set to obtain a global information video frame data set.

[0073] Generally, in video compression, each frame represents a still image. In actual compression, various algorithms are used to reduce the data capacity. The following takes the IPB (Inter-frame Compression Coding Format) compression algorithm as an example to explain in detail how to strip key frames.

[0074] For the IPB compression algorithm, the I frame is a key frame and belongs to intra-frame compression, which is the same as the compression of AVI (Audio Video Interleaved). P means forward search. B means bidirectional search. Both P and B compress data based on I frames. Specifically, the I frame represents the key frame, that is, the complete preservation of this frame, and only the data of this frame is needed for decoding; the P frame represents the difference between this frame and the previous key frame (or P frame). When decoding, the difference defined by this frame needs to be superimposed with the previously cached picture to generate the final picture, that is, the P frame does not have complete picture data, but only the data of the difference with the previous frame; the B frame is a bidirectional difference frame, that is, the B frame records the difference between this frame and the previous and next frames.

[0075] That is to say, when stripping key frames, it is only necessary to strip out the I frames therein. How to determine the I frames specifically can be determined according to the specific definition of the IPB compression algorithm, which will not be described in detail here.

[0076] Preferably, step S200 extracts sample key frames from the video in the video data set to obtain a global information video frame data set, including:

[0077] Step 1: Extract key frames from the videos in the video dataset;

[0078] Step 2: After edge-filling processing is performed on the key frame, bilinear interpolation scaling is performed on the key frame according to a preset size to obtain a global information video frame.

[0079] In order to avoid deformation of the content in the image after the size transformation, after the key frame is extracted, the key frame may be subjected to edge infill processing. Specifically, the key frame may be processed using an existing edge infill solution.

[0080] After completing the edge filling process, the key frame can be scaled according to the preset size. Specifically, the key frame can be scaled by bilinear interpolation scaling to obtain a global information video frame. Of course, in actual applications, when performing size scaling, other scaling methods can be used to further normalize the size of the video frame, which will not be explained here one by one.

[0081] S300 , local information in a sample key frame is located and cropped to obtain a local information video frame data set.

[0082] Among them, local information refers to partial information relative to global information.

[0083] The local information can be found in the global information video frame, and then the local information can be cropped to obtain the local information video frame. For example, the global information video frame corresponds to an N*M image, and the local information video frame can be a local area in the image, with a size of n*m (n is less than or equal to N, m is less than or equal to M, and n and m are different and are equal to N and M respectively).

[0084] Preferably, step S300 locates local information in the sample key frame and performs cropping to obtain a local information video frame data set, including:

[0085] Step 1: Detect the abnormal points in the global information video frame and determine the abnormal points as the local information center points;

[0086] Step 2: Use the outliers to crop the global information video frame to obtain the local information video frame.

[0087] That is to say, an image anomaly detection algorithm can be used to detect abnormal points in the global information video frame, such as points whose brightness differs from adjacent pixels by more than a preset threshold, and points whose color saturation differs from adjacent pixels by more than a preset threshold.

[0088] After the outlier point is determined, the outlier point can be directly determined as the center point with information. That is, a rectangular frame with a specific scale and the center being the outlier point can be used to crop the image based on the outlier degree, thereby obtaining a video frame with information.

[0089] S400: Determine a global information video frame dataset and a local information video frame dataset as low-quality video datasets.

[0090] S500: Obtain an initial low-quality video attribution recognition model.

[0091] The initial low-quality video attribution recognition model can be a newly constructed model, or it can be an initial low-quality video attribution recognition model obtained after the first step of fine-tuning by adopting the idea of ​​transferring information and training the model in a two-step fine-tuning manner. For example, the initial low-quality video attribution recognition model can be a pre-trained model fine-tuning network classifier that is first used on the public dataset of ImageNet (a large visual database for visual object recognition software research).

[0092] S600: Input samples of the low-quality video data set into an initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples.

[0093] S700: Based on the attribution recognition results and corresponding labels of the samples, adjust the initial low-quality video attribution recognition model to obtain a trained low-quality video attribution recognition model.

[0094] Wherein, step S700 adjusts the initial low-quality video attribution recognition model based on the attribution recognition result and the corresponding label of the sample to obtain a trained low-quality video attribution recognition model, including:

[0095] Step 1: Compare the attribution recognition result corresponding to the sample with the corresponding label to obtain a comparison result; wherein the comparison result is the corresponding low-quality video attribution recognition accuracy rate;

[0096] Step 2: Calculate the model loss value using the low-quality video attribution recognition accuracy;

[0097] Step 3: When the model loss value reaches a preset standard, stop iterating and adjusting the initial low-quality video attribution recognition model to obtain a low-quality video attribution recognition model.

[0098] Specifically, the accuracy of low-quality video attribution recognition can be determined based on the similarity between the attribution recognition result and the corresponding label, and then the model loss value is calculated using the low-quality video attribution recognition accuracy. The initial low-quality video attribution recognition model is iteratively adjusted based on the model loss value, and finally a low-quality video attribution recognition model that meets the preset standard is obtained. The preset standard may be that the model loss value tends to be stable or reaches a certain preset value, or the number of iterative training reaches a preset number.

[0099] In a specific implementation of the present application, step S700 inputs a sample of a low-quality video data set into an initial low-quality video attribution recognition model to obtain an attribution recognition result corresponding to the sample, specifically including:

[0100] Step 1: taking the global information video frame and the local information video frame as inputs of a two-way network of an initial low-quality video attribution recognition model;

[0101] Step 2: extracting the global features of the global information video frame and the local features of the local information video frame respectively;

[0102] Specifically, the feature extraction layer in the low-quality video attribution recognition model can be used to extract global features of the global information video frame and local features of the local information video frame.

[0103] Step 3: Perform feature mapping on the global features and local features along the channel dimension to obtain a global feature map and a local feature map with the same dimension;

[0104] Specifically, the last convolution module of the feature extraction layer can be used to perform feature mapping on the global features and the local features along the channel dimension to obtain a global feature map and a local feature map with the same dimension.

[0105] Step 4: Fuse the global feature map and the local feature map to obtain a mixed bilinear feature map;

[0106] Specifically, the implementation process includes:

[0107] Step 1: Pool the global feature map and the local feature map to obtain a global feature pooling vector and a local feature pooling vector;

[0108] Step 2 performs point-to-point inner product processing on the global feature vector and the local feature pooling vector to obtain a mixed bilinear feature map.

[0109] Step 5: Use the hybrid bilinear feature map to obtain the attribution recognition results corresponding to the samples in the low-quality video dataset.

[0110] Specifically, the implementation process includes:

[0111] Step 1: Pool the mixed bilinear feature map to obtain a mixed feature vector;

[0112] Step 2: Connect the mixed feature vector to the fully connected layer, and output the category probability distribution of each video low-quality category through a normalized exponential function;

[0113] Step 3: Use the category probability distribution to determine the attribution recognition result.

[0114] Apply the method provided in the embodiment of the present application to obtain a labeled video dataset in a target business scenario; extract sample key frames from the video in the video dataset to obtain a global information video frame dataset; locate local information in the sample key frames and crop them to obtain a local information video frame dataset; determine the global information video frame dataset and the local information video frame dataset as low-quality video datasets; obtain an initial low-quality video attribution recognition model; input samples of the low-quality video dataset into the initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples; based on the attribution recognition results and corresponding labels of the samples, adjust the initial low-quality video attribution recognition model to obtain a trained low-quality video attribution recognition model.

[0115] In the present application, a labeled video dataset in the target business scenario is first obtained, and then sample key frames are extracted from the videos in the video dataset to obtain a global information video frame dataset. Then, the local information of the global information video frame is located and cropped to obtain a local information video frame dataset. The global information video frame dataset and the local information video frame dataset are determined as low-quality video datasets, and samples of the low-quality video dataset are input into the initial low-quality video attribution recognition model to obtain the attribution recognition results corresponding to the samples. Based on the attribution recognition results and corresponding labels of the samples, the initial low-quality video attribution recognition model is adjusted to obtain a trained low-quality video attribution recognition model. Since the training of the low-quality video attribution recognition model not only pays attention to global features but also local features, it can effectively improve the accuracy of the attribution recognition results of the low-quality video attribution recognition model finally trained. Furthermore, the video quality improvement suggestions can be made more accurate and more conducive to improving video quality.

[0116] In this application, a video processing method is also provided. The video processing method has similar or identical steps to the model training method and can be referenced to each other. Figure 2 , Figure 2 This is a flowchart of a video processing method in an embodiment of the present application, the method comprising the following steps:

[0117] S101, acquiring a target video, and extracting key frames from the target video to obtain a target global information video frame.

[0118] The target video may be any video that needs to be identified for low video quality attribution, and the video may be a live video or a recorded video.

[0119] In this embodiment, the target video can be obtained by directly recording, by reading a disk, or by transmitting the target video over a network.

[0120] After obtaining the target video, the key frames in the target video can be extracted to obtain the target global information video frame. That is, the key frames can be directly determined as the target global information video frame, or the target global information video frame can be obtained by preprocessing the key frames.

[0121] Preferably, in order to facilitate model processing, the size of the target global information video frame can also be controlled within a preset size. That is, in practical applications, extracting key frames from the target video in step S101 to obtain the target global information video frame can specifically include:

[0122] Step 1: Extract key frames from the target video;

[0123] Step 2: After edge-filling processing is performed on the key frame, bilinear interpolation scaling is performed on the key frame according to a preset size to obtain a target global information video frame.

[0124] For ease of description, the above two steps are combined for explanation below.

[0125] In order to avoid deformation of the content in the image after the size transformation, after the key frame is extracted, the key frame may be subjected to edge infill processing. Specifically, the key frame may be processed using an existing edge infill solution.

[0126] After completing the edge filling process, the key frame can be scaled according to the preset size. Specifically, the key frame can be scaled by bilinear interpolation scaling to obtain the target global information video frame. Of course, in actual applications, when performing size scaling, other scaling methods can be used to further normalize the size of the video frame, which will not be explained here one by one.

[0127] S102: Locate local information of a target global information video frame and crop it to obtain a target local information video frame.

[0128] Among them, local information refers to partial information relative to global information.

[0129] That is, local information can be found in the target global information video frame, and then the local information can be cropped to obtain the target local information video frame. For example, the target global information video frame corresponds to an N*M image, and the target local information video frame can be a local area in the image, with a size of n*m (n is less than or equal to N, m is less than or equal to M, and n and m are not equal to N and M respectively).

[0130] Preferably, in order to make the target local information video frame more representative, when locating the local information, it can be located based on the abnormal point. Specifically, step S102 locates the local information of the target global information video frame and crops it to obtain the target local information video frame, including:

[0131] Step 1: Detect the abnormal points in the target global information video frame and determine the abnormal points as the local information center points;

[0132] Step 2: Use the outliers to crop the target global information video frame to obtain the target local information video frame.

[0133] For ease of description, the above two steps are combined for explanation below.

[0134] An image anomaly detection algorithm can be used to detect abnormal points in the target global information video frame, such as points whose brightness differs from adjacent pixels by more than a preset threshold, and points whose color saturation differs from adjacent pixels by more than a preset threshold.

[0135] After the outlier point is determined, the outlier point can be directly determined as the center point with information. That is, a rectangular frame with a specific scale and the center being the outlier point can be used to crop the image based on the outlier degree, thereby obtaining a video frame with information.

[0136] S103, inputting the target global information video frame and the target local information video frame into the low-quality video attribution recognition model trained corresponding to the above-mentioned model training method to perform low-quality video attribution recognition, and obtaining the target attribution recognition result.

[0137] In an embodiment of the present application, a low-quality video attribution recognition model can be pre-trained, and the model can perform attribution recognition on low-quality videos based on target global information video frames and target local information video frames.

[0138] That is to say, in this embodiment, the low-quality video attribution recognition model has two inputs, one is the target global information video frame, and the other is the target local information video frame.

[0139] The low-quality video attribution recognition model can extract features from the two inputs and combine the two features to perform low-quality attribution recognition, thereby obtaining the target attribution recognition result.

[0140] Specifically, the target attribution recognition result can specifically display the specific cause type that causes the low quality of the target video, such as overexposure, noise, masking, stray light, normal, unknown, etc. The target attribution recognition result can correspond to only one specific cause, or can be a probability ranking of each cause.

[0141] S104: Based on the target attribution recognition result, output video quality improvement suggestions.

[0142] After the model outputs the target attribution recognition result, it can output the video quality improvement suggestion based on the target attribution recognition result. Specifically, the specific reasons leading to low video quality can be bound to the video quality extraction suggestion one by one in advance. In this way, after obtaining the target attribution recognition result, the corresponding video quality improvement suggestion can be quickly determined based on the binding relationship and output, so that the user can improve the video quality based on the suggestion.

[0143] Apply the method provided in the embodiment of the present application to obtain a target video, extract key frames from the target video, and obtain a target global information video frame; locate the local information of the target global information video frame and crop it to obtain a target local information video frame; input the target global information video frame and the target local information video frame into a low-quality video attribution recognition model trained corresponding to the above-mentioned model training method to perform low-quality video attribution recognition to obtain a target attribution recognition result; based on the target attribution recognition result, output a video quality improvement suggestion.

[0144] In the present application, after acquiring the target video, the key frame is first extracted to obtain the target global information video frame. Then, the local information of the target global information video frame is located and cropped to obtain the target local information video frame. Then, based on the trained low-quality video attribution recognition model, low-quality video attribution recognition is performed on the target global information video frame and the target local information video frame, and finally the target attribution recognition result is obtained. Based on the target attribution recognition result, a video quality improvement suggestion is output so that the user can extract suggestions to improve the video quality based on the video instruction. In other words, the input to the low-quality video attribution recognition model includes the target global information video frame and the target local information video frame. Therefore, in the process of model processing, not only the global features but also the local features are paid attention to. Combining the two for attribution recognition can effectively improve the accuracy of the target attribution recognition result. In this way, the output video quality improvement suggestion can be more accurate and more conducive to improving the video quality.

[0145] It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the preferred / improved embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be described one by one in the preferred / improved embodiments of this article.

[0146] In a specific implementation of the present application, the above step S103 inputs the target global information video frame and the target local information video frame into the low-quality video attribution recognition model trained by the above model training method to perform low-quality video attribution recognition, and obtains the target attribution recognition result, including:

[0147] Step 1: Use the target global information video frame and the target local information video frame as inputs of the dual-path network of the low-quality video attribution recognition model.

[0148] Step 2: Use the feature extraction layer in the low-quality video attribution recognition model to extract the global features of the target global information video frame and the local features of the target local information video frame.

[0149] Step 3: Use the last convolution module of the feature extraction layer to perform feature mapping on the global features and local features along the channel dimension to obtain global feature maps and local feature maps with the same dimension.

[0150] Step 4: Fuse the global feature map and the local feature map to obtain a mixed bilinear feature map.

[0151] Specifically, step four may include:

[0152] Step 1: Pool the global feature map and the local feature map to obtain a global feature pooling vector and a local feature pooling vector;

[0153] Step 2: Perform point-to-point inner product processing on the global feature vector and the local feature pooling vector to obtain a mixed bilinear feature map.

[0154] Step 5: Use the mixed bilinear feature map to determine the target attribution recognition result.

[0155] Specifically, step five may include:

[0156] Step 1: Pool the mixed bilinear feature map to obtain a mixed feature vector;

[0157] Step 2: Connect the mixed feature vector to the fully connected layer, and output the category probability distribution of each video low-quality category through a normalized exponential function;

[0158] Step 3: Use the category probability distribution to determine the target attribution recognition result.

[0159] For ease of explanation, the above five steps are combined and explained below.

[0160] That is, the target global information video frame and the corresponding target local information video frame are used as the input of the two-way network respectively. Preferably, taking into account the network feature extraction capability and the model feedforward speed, ResNet-50 is used as the basic network of the low-quality video attribution recognition architecture, and the two-way network shares the same initialization weights to extract the global and local features of the video frame respectively, wherein the global features focus more on the overall image quality of the video frame, and the local features focus more on the local texture quality of the video frame.

[0161] Specifically, the last convolution module of the dual-path network feature extraction layer can be selected to perform feature mapping along the channel dimension to obtain a feature map with a higher dimension, so that the global feature map after mapping maintains the same dimension as the local feature map. Perform a summing and pooling operation on the local feature mapping layer (of course, the maximum pooling operation can also be selected) to obtain a local feature pooling vector, the dimension of which is the same as the channel dimension of the global feature mapping layer. Perform inner product operations one by one on the local feature pooling vector and the feature vector along the channel direction on the global feature mapping layer to obtain a mixed bilinear feature map, and perform summing and pooling on the mixed bilinear pooling feature map to obtain the final mixed feature vector.

[0162] The mixed feature vector obtained after summing and pooling is connected to the fully connected layer, and the category probability distribution of each low-quality category is output through the Softmax (normalized exponential) function. The category with the largest probability value is the final low-quality prediction category.

[0163] In a specific implementation of the present application, in order to improve the model training efficiency, the idea of ​​transferring information can also be adopted, and the model can be trained based on a two-step fine-tuning method. Specifically, the process of training the low-quality video attribution recognition model includes:

[0164] Step 1: Obtain a low-quality video dataset;

[0165] Step 2: Fine-tune the network classifier using the pre-trained model on the ImageNet public dataset;

[0166] Step 3: Use the fine-tuned model as the initialization parameter of the entire network, and fine-tune the entire network in combination with the low-quality video dataset to obtain a low-quality video attribution recognition model whose low-quality video attribution recognition accuracy reaches the preset standard.

[0167] In other words, we can adopt the idea of ​​transfer learning and a two-step fine-tuning training method. First, we use the pre-trained model on the ImageNet (a large visual database for visual object recognition software research) public dataset to fine-tune the network classifier. Then, we use the model fine-tuned in the previous step as the initialization parameter of the entire network to perform tuning training on the entire network. We use the cross-entropy loss function and the low-quality video attribution recognition accuracy as the effectiveness measurement indicator of the model.

[0168] The network's hyperparameters are adjusted to obtain parameter weights with relatively better effects as the final network model. The accuracy results of the model are compared with those of existing technical solutions and combined with subjective perception.

[0169] Preferably, the model can also be compressed to reduce network parameters, improve feedforward speed, and make the model performance better. When deploying, the optimal model can be selected for engineering deployment, and ultimately achieve the purpose of real-time low-quality recognition online.

[0170] Considering that in this application, it is proposed to perform low-quality video attribution recognition based on global information and local information, when training the model, it is also necessary to use samples with global information and samples with local information for model training. The following takes the live video scene as an example to explain in detail how to obtain a low-quality video dataset that can meet this requirement. Specifically, the above step 1 obtains a low-quality video dataset, including:

[0171] Step 1: Build a diverse index video dataset for live broadcast business scenarios, and add low-quality live broadcast video samples to obtain a low-quality live broadcast video dataset;

[0172] Step 2: Label the video samples in the low-quality live video dataset with classification labels, and filter out the video samples that do not meet the preset conditions to obtain a labeled video dataset;

[0173] Step 3: extract sample key frames from the video in the video dataset to obtain a target global information video frame dataset;

[0174] Step 4: locate the local information in the sample key frame and crop it to obtain the target local information video frame dataset;

[0175] Step 5: Determine the target global information video frame dataset and the target local information video frame dataset as low-quality video datasets.

[0176] That is, first, we can build a diversified live video data set for the live broadcast business scenario, manually screen and add low-quality live video samples in batches, and perform data distribution balancing to make the low-quality video reach an appropriate proportion (such as 0.5, of course, it can also be other appropriate ratios, which can be obtained through practical data statistics, and will not be listed here one by one).

[0177] Then, the low-quality live video dataset can be screened again to filter out video samples that do not meet the requirements (such as low quality caused by network transmission, which is difficult for the video recorder to improve), and objective classification and labeling are performed at the same time. The label types include overexposure, noise, masking, stray light, normal, unknown, etc., to form a labeled video dataset.

[0178] Finally, we use traditional image processing tools and algorithms to extract key frame images from the video, perform edge complementation operations on the key frames, and perform bilinear interpolation scaling according to the given size to obtain the target global information video frame dataset. We locate the local information in the frame image and crop it to obtain the target local information video frame dataset. We divide the training set, validation set, and test set into proportions.

[0179] In order to facilitate those skilled in the art to better understand the video processing method provided in the embodiment of the present application, the following takes attribution identification of low-quality live videos in a live broadcast scenario as an example to explain in detail how to implement the video processing method.

[0180] Please refer to Figure 3 and Figure 4 , Figure 3 Schematic diagram of an implementation of low-quality video attribution identification in an embodiment of the present application. Figure 4 Schematic diagram of a flow chart of a low-quality video attribution identification method in an embodiment of the present application.

[0181] Among them, the construction, data cleaning and preprocessing of low-quality live broadcast data specifically include: data set construction link and data cleaning and preprocessing link. The data set construction link includes: data collection, data initial screening, random grouping, low-quality video sampling, separate balanced processing, labeling rules formulation, objective labeling and finally obtaining a labeled data set. The data cleaning and preprocessing links include: label review, video key frame extraction, bilinear interpolation scaling, global information data set, local information positioning cropping, local information data set, data set division (such as training set, validation set and test set).

[0182] The end-to-end architecture design based on the hybrid bilinear hybrid pooling model includes: end-to-end network construction and model training. The end-to-end network construction includes: dual-path information input, basic network selection, dual-path feature extraction, high-dimensional feature mapping, hybrid bilinear pooling, hybrid feature representation, Softmax and attribution recognition (attribution recognition can also refer to Figure 4 ). Model training includes: data augmentation, loss function, optimizer, pre-trained model, learning rate decay strategy, full network training, regularization, hyperparameter tuning, and final model.

[0183] The establishment of the evaluation plan, accuracy assessment and tuning can be divided into model indicators and effect evaluation, model tuning and engineering. Among them, model indicators and effect evaluation include: indicator evaluation (such as loss, accuracy), cross-validation, parameter calculation and effect evaluation (SOTA (state-of-the-art, model performance is currently the best comparison algorithm) comparison and subjective perception). Model tuning and engineering include: performance tuning (such as decoding optimization, feedforward optimization and indicator optimization), model compression (such as knowledge distillation and model pruning), engineering (such as C++ reconstruction, background deployment and online operation).

[0184] It should be noted that the corresponding relevant terms in the specific implementation, such as cross-validation, knowledge distillation, etc., can be specifically referred to the relevant implementations and definitions, and will not be explained one by one here.

[0185] exist Figure 4 In the figure, the upper left image corresponds to the target local information video frame corresponding to the middle video frame, and the lower left image corresponds to the target global information video frame corresponding to the middle video frame. The process of analyzing the video frame can be specifically referred to the above description, and will not be repeated here.

[0186] It can be seen that the video processing method provided in the embodiment of the present application can be used for attribution identification of low-quality live video, and can provide real-time recording suggestions for anchors with relatively low live broadcast quality, thereby improving the quality of live broadcast viewing and bringing a better live broadcast experience to both anchors and users.

[0187] Corresponding to the above method embodiment, the embodiment of the present application also provides a model training device, and the model training device described below and the model training method described above can refer to each other.

[0188] See also Figure 5 As shown, the device includes the following modules:

[0189] The data set acquisition module 100 is used to acquire a labeled video data set in a target business scenario;

[0190] A global information acquisition module 200 is used to extract sample key frames from the video in the video data set to obtain a global information video frame data set;

[0191] A local information acquisition module 300 is used to locate and crop the local information in the sample key frame to obtain a local information video frame data set;

[0192] A low-quality video data set 400 is used to determine the global information video frame data set and the local information video frame data set as a low-quality video data set;

[0193] An initial model acquisition module 500 is used to acquire an initial low-quality video attribution recognition model;

[0194] An attribution recognition module 600 is used to input samples of the low-quality video data set into an initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples;

[0195] The training adjustment module 700 is used to adjust the initial low-quality video attribution recognition model based on the attribution recognition results and corresponding labels of the samples to obtain a trained low-quality video attribution recognition model.

[0196] By using the device provided in the embodiment of the present application, a labeled video dataset in a target business scenario is obtained; sample key frames are extracted from the video in the video dataset to obtain a global information video frame dataset; local information in the sample key frames is located and cropped to obtain a local information video frame dataset; the global information video frame dataset and the local information video frame dataset are determined as low-quality video datasets; an initial low-quality video attribution recognition model is obtained; samples of the low-quality video dataset are input into the initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples; based on the attribution recognition results and corresponding labels of the samples, the initial low-quality video attribution recognition model is adjusted to obtain a trained low-quality video attribution recognition model.

[0197] In the present application, a labeled video dataset in the target business scenario is first obtained, and then sample key frames are extracted from the videos in the video dataset to obtain a global information video frame dataset. Then, the local information of the global information video frame is located and cropped to obtain a local information video frame dataset. The global information video frame dataset and the local information video frame dataset are determined as low-quality video datasets, and samples of the low-quality video dataset are input into the initial low-quality video attribution recognition model to obtain the attribution recognition results corresponding to the samples. Based on the attribution recognition results and corresponding labels of the samples, the initial low-quality video attribution recognition model is adjusted to obtain a trained low-quality video attribution recognition model. Since the training of the low-quality video attribution recognition model not only pays attention to global features but also local features, it can effectively improve the accuracy of the attribution recognition results of the low-quality video attribution recognition model finally trained. Furthermore, the video quality improvement suggestions can be made more accurate and more conducive to improving video quality.

[0198] In a specific implementation of the present application, the training adjustment module 700 is specifically used to:

[0199] Compare the attribution recognition result corresponding to the sample with the corresponding label to obtain a comparison result; wherein the comparison result is the corresponding low-quality video attribution recognition accuracy rate;

[0200] The model loss value is calculated using the attribution recognition accuracy of low-quality videos;

[0201] When the model loss value reaches a preset standard, the iterative adjustment of the initial low-quality video attribution recognition model is stopped to obtain a low-quality video attribution recognition model.

[0202] In a specific implementation of the present application, the data set acquisition module 100 is specifically used to:

[0203] Build a diverse index video dataset for live broadcast business scenarios and add low-quality live broadcast video samples to obtain a low-quality live broadcast video dataset;

[0204] The video samples in the low-quality live video dataset are labeled with classification labels, and the video samples that do not meet the preset conditions are filtered out to obtain a labeled video dataset.

[0205] In a specific implementation of the present application, the global information acquisition module 200 is specifically used to:

[0206] Extract key frames from videos in a video dataset;

[0207] After edge filling processing is performed on the key frame, bilinear interpolation scaling is performed on the key frame according to a preset size to obtain a global information video frame.

[0208] In a specific implementation of the present application, the local information acquisition module 300 is specifically used to:

[0209] Detecting abnormal points in the global information video frame and determining the abnormal points as the local information center points;

[0210] The global information video frame is cropped using the outliers to obtain the local information video frame.

[0211] In a specific implementation of the present application, the attribution identification module 600 is specifically used to:

[0212] The global information video frame and the local information video frame are used as inputs of a two-way network of an initial low-quality video attribution recognition model;

[0213] Extracting the global features of the global information video frame and the local features of the local information video frame respectively;

[0214] Perform feature mapping on the global features and local features along the channel dimension to obtain global feature maps and local feature maps with the same dimension;

[0215] The global feature map and the local feature map are fused to obtain a mixed bilinear feature map;

[0216] By using the hybrid bilinear feature map, the attribution recognition results corresponding to the samples in the low-quality video dataset are obtained.

[0217] In a specific implementation of the present application, the attribution identification module 600 is specifically used to:

[0218] Pooling the global feature map and the local feature map to obtain a global feature pooling vector and a local feature pooling vector;

[0219] The global feature vector and the local feature pooling vector are processed point-to-point by inner product to obtain a mixed bilinear feature map.

[0220] In a specific implementation of the present application, the attribution identification module 600 is specifically used to:

[0221] Pooling the mixed bilinear feature map to obtain a mixed feature vector;

[0222] Connect the mixed feature vector to the fully connected layer and output the category probability distribution of each video low quality category through the normalized exponential function;

[0223] The attribution recognition results are determined using the category probability distribution.

[0224] Corresponding to the above method embodiment, the embodiment of the present application further provides a video processing device. The video processing device described below and the video processing method described above can refer to each other.

[0225] See also Figure 6 As shown, the device includes the following modules:

[0226] The global information video frame acquisition module 101 is used to acquire a target video and extract key frames from the target video to obtain a target global information video frame;

[0227] The local information video frame acquisition module 102 is used to locate the local information of the target global information video frame and crop it to obtain the target local information video frame;

[0228] The attribution recognition module 103 is used to input the target global information video frame and the target local information video frame into the low-quality video attribution recognition model trained by the above-mentioned model training method to perform low-quality video attribution recognition and obtain a target attribution recognition result;

[0229] The suggestion output module 104 is used to output video quality improvement suggestions based on the target attribution recognition result.

[0230] By using the device provided in the embodiment of the present application, a target video is acquired, and key frames of the target video are extracted to obtain a target global information video frame; local information of the target global information video frame is located and cropped to obtain a target local information video frame; the target global information video frame and the target local information video frame are input into a low-quality video attribution recognition model trained corresponding to the above-mentioned model training method to perform low-quality video attribution recognition to obtain a target attribution recognition result; based on the target attribution recognition result, a video quality improvement suggestion is output.

[0231] In the present application, after acquiring the target video, the key frame is first extracted to obtain the target global information video frame. Then, the local information of the target global information video frame is located and cropped to obtain the target local information video frame. Then, based on the trained low-quality video attribution recognition model, low-quality video attribution recognition is performed on the target global information video frame and the target local information video frame, and finally the target attribution recognition result is obtained. Based on the target attribution recognition result, a video quality improvement suggestion is output so that the user can extract suggestions to improve the video quality based on the video instruction. In other words, the input to the low-quality video attribution recognition model includes the target global information video frame and the target local information video frame. Therefore, in the process of model processing, not only the global features but also the local features are paid attention to. Combining the two for attribution recognition can effectively improve the accuracy of the target attribution recognition result. In this way, the output video quality improvement suggestion can be more accurate and more conducive to improving the video quality.

[0232] In a specific implementation of the present application, the attribution recognition module 103 is specifically configured to use the target global information video frame and the target local information video frame as inputs of a two-way network of a low-quality video attribution recognition model;

[0233] The feature extraction layer in the low-quality video attribution recognition model is used to extract the global features of the target global information video frame and the local features of the target local information video frame respectively;

[0234] Using the last convolutional module of the feature extraction layer, feature mapping is performed on the global features and local features along the channel dimension to obtain global feature maps and local feature maps with the same dimension.

[0235] The global feature map and the local feature map are fused to obtain a mixed bilinear feature map;

[0236] The attribution recognition result is determined by using the hybrid bilinear feature map.

[0237] In a specific implementation of the present application, the attribution identification module 103 is specifically used to pool the global feature map and the local feature map to obtain a global feature pooling vector and a local feature pooling vector;

[0238] The global feature vector and the local feature pooling vector are processed point-to-point by inner product to obtain a mixed bilinear feature map.

[0239] In a specific implementation of the present application, the attribution identification module 103 is specifically used to pool the mixed bilinear feature map to obtain a mixed feature vector;

[0240] Connect the mixed feature vector to the fully connected layer and output the category probability distribution of each video low quality category through the normalized exponential function;

[0241] The attribution recognition results are determined using the category probability distribution.

[0242] In a specific implementation of the present application, the global information video frame acquisition module 101 is specifically used to extract key frames from the target video;

[0243] After edge filling processing is performed on the key frame, bilinear interpolation scaling is performed on the key frame according to a preset size to obtain a target global information video frame.

[0244] In a specific implementation of the present application, the local information video frame acquisition module 102 is specifically used to detect abnormal points in the target global information video frame and determine the abnormal points as local information center points;

[0245] The target global information video frame is cropped using the outliers to obtain the target local information video frame.

[0246] In a specific implementation of the present application, it also includes:

[0247] A data set acquisition module, used to acquire low-quality video data sets;

[0248] The training module is used to fine-tune the network classifier based on the pre-trained model on the ImageNet public dataset;

[0249] The fine-tuned model is used as the initialization parameter of the entire network, and the entire network is tuned and trained in combination with a low-quality video dataset to obtain a low-quality video attribution recognition model with an accuracy rate that reaches a preset standard.

[0250] In a specific implementation of the present application, the data set acquisition module is specifically used to construct a diversified indicator video data set for the live broadcast business scenario, and add low-quality live broadcast video samples to obtain a low-quality live broadcast video data set;

[0251] Label the video samples in the low-quality live video dataset with classification labels, and filter out the video samples that do not meet the preset conditions to obtain a labeled video dataset;

[0252] Extract sample key frames from the video in the video dataset to obtain a target global information video frame dataset;

[0253] Locate the local information in the sample key frame and crop it to obtain the target local information video frame dataset;

[0254] The target global information video frame dataset and the target local information video frame dataset are determined as low-quality video datasets.

[0255] Corresponding to the above method embodiment, an embodiment of the present application further provides an electronic device. The electronic device described below and the video processing method described above can refer to each other.

[0256] See also Figure 7 As shown, the electronic device includes:

[0257] A memory 332, for storing computer programs;

[0258] The processor 322 is used to implement the steps of the video processing method of the above method embodiment when executing a computer program.

[0259] For details, please refer to Figure 8 , Figure 8 A specific structural diagram of an electronic device provided for this embodiment, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer applications 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the central processing unit 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.

[0260] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0261] The steps in the video processing method described above can be implemented by the structure of an electronic device.

[0262] Corresponding to the above method embodiment, the embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the video processing method described above can refer to each other.

[0263] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video processing method of the above method embodiment are implemented.

[0264] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0265] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. A model training method, characterized in that: include: Obtain a labeled video dataset for the target business scenario; Extracting sample key frames from the videos in the video data set to obtain a global information video frame data set; Locating the local information in the sample key frame and cropping it to obtain a local information video frame data set; Determining the global information video frame dataset and the local information video frame dataset as low-quality video datasets; Obtain an initial low-quality video attribution recognition model; Inputting samples of the low-quality video data set into the initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples; Based on the attribution recognition result of the sample and the corresponding label, adjusting the initial low-quality video attribution recognition model to obtain a trained low-quality video attribution recognition model; The local information in the sample key frame is located and cropped to obtain a local information video frame data set, including: Detecting an abnormal point in the global information video frame, and determining the abnormal point as the local information center point; The global information video frame is cropped using the abnormal points to obtain the local information video frame.

2. The model training method according to claim 1, characterized in that: Based on the attribution recognition result of the sample and the corresponding label, adjusting the initial low-quality video attribution recognition model to obtain a trained low-quality video attribution recognition model, including: Comparing the attribution recognition result corresponding to the sample with the corresponding label to obtain a comparison result; wherein the comparison result is the corresponding low-quality video attribution recognition accuracy rate; Calculate the model loss value using the low-quality video attribution recognition accuracy; When the model loss value reaches a preset standard, the iterative adjustment of the initial low-quality video attribution recognition model is stopped to obtain the low-quality video attribution recognition model.

3. The model training method according to claim 1, characterized in that: The step of obtaining a video dataset with labels in the target business scenario includes: Build a diverse index video dataset for live broadcast business scenarios and add low-quality live broadcast video samples to obtain a low-quality live broadcast video dataset; Classification labels are marked for video samples in the low-quality live video data set, and video samples that do not meet preset conditions are filtered to obtain the labeled video data set.

4. The model training method according to claim 1, characterized in that: Extracting sample key frames from the video in the video data set to obtain a global information video frame data set, including: Extracting key frames from the videos in the video dataset; After edge filling processing is performed on the key frame, bilinear interpolation scaling is performed on the key frame according to a preset size to obtain the global information video frame.

5. The model training method according to any one of claims 1 to 4, characterized in that: Inputting samples of the low-quality video data set into the initial low-quality video attribution recognition model to obtain attribution recognition results corresponding to the samples, including: Using the global information video frame and the local information video frame as inputs of a two-way network of the initial low-quality video attribution recognition model; Respectively extracting global features of the global information video frame and local features of the local information video frame; Performing feature mapping on the global features and the local features along the channel dimension to obtain a global feature map and a local feature map with the same dimension; Fusion of the global feature map and the local feature map to obtain a mixed bilinear feature map; The mixed bilinear feature map is used to obtain attribution recognition results corresponding to samples in the low-quality video data set.

6. The model training method according to claim 5, characterized in that: The global feature map and the local feature map are fused to obtain a mixed bilinear feature map, including: Pooling the global feature map and the local feature map to obtain a global feature pooling vector and a local feature pooling vector; Performing point-to-point inner product processing on the global feature vector and the local feature pooling vector to obtain the mixed bilinear feature map.

7. The model training method according to claim 5, characterized in that: Using the hybrid bilinear feature map, obtaining attribution recognition results corresponding to samples in the low-quality video data set, including: Pooling the mixed bilinear feature map to obtain a mixed feature vector; Connecting the mixed feature vector to a fully connected layer, and outputting the category probability distribution of each video low-quality category through a normalized exponential function; The attribution recognition result is determined using the category probability distribution.

8. A video processing method, characterized in that: include: Acquire a target video, and extract key frames from the target video to obtain a target global information video frame; Locating local information of the target global information video frame and cropping it to obtain a target local information video frame; Inputting the target global information video frame and the target local information video frame into a low-quality video attribution recognition model trained corresponding to the model training method according to any one of claims 1 to 7 to perform low-quality video attribution recognition, and obtaining a target attribution recognition result; Based on the target attribution recognition result, a video quality improvement suggestion is output.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the model training method as described in any one of claims 1 to 7 and / or the steps of the video processing method as described in claim 8 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the model training method according to any one of claims 1 to 7, and / or the steps of the video processing method according to claim 8.

Images