Method and device for evaluating video highlights

Abstract

The present invention provides a method and device for evaluating video highlight segments, relating to the technical field of video image processing. The method comprises: dividing a video to be processed into N segments to be evaluated, where N is an integer greater than 1; determining motion information corresponding to each segment to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each segment to be evaluated; determining texture information corresponding to each segment to be evaluated based on pixel differences between adjacent pixels within each image frame in each segment to be evaluated; determining a highlight value corresponding to each segment to be evaluated based on the motion information and the texture information; wherein the highlight value is used to characterize the highlight level of each segment to be evaluated; and determining at least one segment to be evaluated from the N segments to be evaluated as a highlight segment of the video to be processed based on the highlight value corresponding to each segment to be evaluated. The present invention can effectively improve the accuracy of evaluating video highlight segments.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a method and device for evaluating video highlights. Background Art

[0002] With the rapid development of network technology and smart mobile devices, a large amount of video information has emerged in front of users. This video information has enriched our lives, but the richness of video information comes at the cost of its unstructured nature. It is urgent to find wonderful video clips from this large amount of video for users to choose and watch.

[0003] At present, video highlights are mainly evaluated through deep learning-based evaluation methods. Specifically, the audio features and image features of the same time segment are extracted, the two are merged into a feature vector, and then a pre-trained model is used for evaluation and scoring.

[0004] However, the difficulty of deep learning-based evaluation methods lies in the collection of training samples and the selection of training models. The evaluation accuracy of video highlights is usually low, which is not conducive to actual business use. Summary of the Invention

[0005] The present invention provides a method and device for evaluating video highlights, which are used to solve the problem in the prior art that the evaluation accuracy of video highlights is generally low, which is not conducive to actual business use.

[0006] The present invention provides a method for evaluating video highlights, comprising:

[0007] Divide the video to be processed into N segments to be evaluated, where N is an integer greater than 1;

[0008] Determining motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated;

[0009] Determining texture information corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each image frame in each of the segments to be evaluated;

[0010] Determining a wonderful value corresponding to each of the to-be-evaluated segments based on the motion information and the texture information; wherein the wonderful value is used to represent the degree of wonderfulness of each of the to-be-evaluated segments;

[0011] Based on the wonderful values ​​corresponding to the segments to be evaluated, at least one segment to be evaluated is determined from the N segments to be evaluated as a wonderful segment of the video to be processed.

[0012] According to a method for evaluating video highlights provided by the present invention, determining motion information corresponding to each segment to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each segment to be evaluated includes:

[0013] Decoding each of the to-be-evaluated segments into at least two image frames;

[0014] Acquire at least two grayscale images corresponding to the at least two image frames;

[0015] Obtaining at least one first pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between corresponding pixels of adjacent grayscale images in the at least two grayscale images;

[0016] Calculate the average value mv of the pixel values ​​corresponding to all the pixel points in each of the first pixel difference maps;

[0017] An average value Mmv of mv corresponding to all first pixel difference maps in each of the segments to be evaluated is calculated as the motion information corresponding to each of the segments to be evaluated.

[0018] According to a method for evaluating video highlights provided by the present invention, before obtaining at least one first pixel difference map corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels in adjacent grayscale images of the at least two grayscale images, the method further includes:

[0019] Scaling the at least two grayscale images in equal proportions to obtain at least two scaled images corresponding to the at least two grayscale images;

[0020] The obtaining, based on pixel differences between corresponding pixels between adjacent grayscale images in the at least two grayscale images, at least one first pixel difference image corresponding to each of the to-be-evaluated segments comprises:

[0021] At least one first pixel difference map corresponding to each of the to-be-evaluated segments is obtained based on pixel differences between corresponding pixel points in adjacent zoomed images in the at least two zoomed images.

[0022] According to a method for evaluating video highlights provided by the present invention, determining texture information corresponding to each segment to be evaluated based on pixel differences between adjacent pixels in each image frame in each segment to be evaluated includes:

[0023] Decoding each of the to-be-evaluated segments into at least one image frame;

[0024] Acquire at least one grayscale image corresponding to the at least one image frame;

[0025] scaling the at least one grayscale image in equal proportion to obtain at least one scaled image corresponding to the at least one grayscale image;

[0026] Obtaining at least one second pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between adjacent pixels in each of the zoomed images;

[0027] Calculate the average value tex of the pixel values ​​corresponding to all the pixel points in each of the second pixel difference images;

[0028] An average value Mtex of tex corresponding to all second pixel difference images in each of the segments to be evaluated is calculated as the texture information corresponding to each of the segments to be evaluated.

[0029] According to a method for evaluating video highlights provided by the present invention, obtaining at least one second pixel difference map corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each of the zoomed images includes:

[0030] Calculating pixel differences between adjacent pixels in a horizontal direction in each of the zoomed images to obtain at least one horizontal difference image corresponding to each of the segments to be evaluated;

[0031] Calculating pixel differences between adjacent pixels in a vertical direction in each of the zoomed images to obtain at least one vertical difference image corresponding to each of the segments to be evaluated;

[0032] At least one second pixel difference map corresponding to each of the to-be-evaluated segments is obtained based on the at least one horizontal difference map and the at least one vertical difference map.

[0033] According to a method for evaluating video highlights provided by the present invention, obtaining at least one second pixel difference map corresponding to each of the segments to be evaluated based on the at least one horizontal difference map and the at least one vertical difference map includes:

[0034] In the horizontal difference map and the vertical difference map corresponding to each zoom map, the maximum pixel value of the corresponding pixel point is selected to form the maximum difference map corresponding to each zoom map as at least one second pixel difference map corresponding to each to-be-evaluated segment.

[0035] According to a method for evaluating video highlights provided by the present invention, the method of dividing a video to be processed into N segments to be evaluated includes:

[0036] Based on the time stamps of the key frames preset in the video to be processed, the video to be processed is divided into the N segments to be evaluated.

[0037] According to a method for evaluating video highlights provided by the present invention, the method of dividing the video to be processed into the N segments to be evaluated based on the timestamps of key frames preset in the video to be processed includes:

[0038] Obtaining the timestamp of a key frame preset in the video to be processed;

[0039] Subtracting the timestamp of the first key frame from the timestamps of the key frames to obtain the timestamp of the target key frame; wherein the first key frame is a key frame whose timestamp difference with an adjacent key frame is less than a first threshold, and the adjacent key frame is a key frame adjacent to the first key frame;

[0040] Based on the timestamp of the target key frame, the video to be processed is divided into the N segments to be evaluated.

[0041] According to a method for evaluating video highlight segments provided by the present invention, determining a highlight value corresponding to each segment to be evaluated based on the motion information and the texture information includes:

[0042] Based on the motion information Mmv and the texture information Mtex, the wonderful value corresponding to each of the segments to be evaluated is determined using formula (1):

[0043] Wonderful value = a*Mmv+(1-a)*Mtex (1)

[0044] Wherein, a is a preset weight value.

[0045] The present invention also provides a device for evaluating video highlights, comprising:

[0046] A segmentation module is used to segment the video to be processed into N segments to be evaluated, where N is an integer greater than 1;

[0047] A first determining module is configured to determine motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated;

[0048] A second determining module is configured to determine texture information corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each image frame in each of the segments to be evaluated;

[0049] A third determining module is configured to determine a wonderful value corresponding to each of the segments to be evaluated based on the motion information and the texture information; wherein the wonderful value is used to represent the wonderfulness of each of the segments to be evaluated;

[0050] The fourth determining module is configured to determine, based on the highlight values ​​corresponding to the respective to-be-evaluated segments, at least one to-be-evaluated segment from the N to-be-evaluated segments as a highlight segment of the to-be-processed video.

[0051] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for evaluating video highlights as described above is implemented.

[0052] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for evaluating video highlights as described above is implemented.

[0053] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for evaluating video highlights.

[0054] The method and device for evaluating video highlight segments provided by the present invention are not limited to the collection of training samples and the selection of training models in comparison with the evaluation method based on deep learning in the related art. The evaluation method provided by the embodiment of the present invention can calculate the highlight value corresponding to the segment to be evaluated based on the texture information and motion information corresponding to the segment to be evaluated divided from the video to be processed, and then determine at least one segment to be evaluated from N segments to be evaluated based on the highlight value, a parameter representing the highlight degree of the segment to be evaluated, as the highlight segment of the video to be processed. Since the calculated highlight value is not affected by subjective factors and can objectively reflect the richness and dynamics of the video scene in the segment to be evaluated, evaluating the segment to be evaluated based on the highlight value can effectively improve the accuracy of evaluating the highlight segments of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 This is one of the flow charts of the method for evaluating video highlights provided by the present invention;

[0057] Figure 2 This is a second flow chart of the method for evaluating video highlights provided by the present invention;

[0058] Figure 3 Schematic diagram of the structure of the device for evaluating video highlights provided by the present invention;

[0059] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] The following describes the method and device for evaluating video highlights of the present invention in conjunction with the accompanying drawings.

[0062] Figure 1 This is one of the flow charts of the method for evaluating video highlights provided by the present invention, such as Figure 1 As shown, the method for evaluating video highlights includes steps 101 to 105; wherein:

[0063] Step 101: Divide the video to be processed into N segments to be evaluated, where N is an integer greater than 1;

[0064] Step 102: determining motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated;

[0065] Step 103: determining texture information corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each image frame in each of the segments to be evaluated;

[0066] Step 104: Determine a wonderful value corresponding to each of the segments to be evaluated based on the motion information and the texture information; wherein the wonderful value is used to represent the wonderfulness of each of the segments to be evaluated;

[0067] Step 105 : Based on the highlight values ​​corresponding to the respective to-be-evaluated segments, determine at least one to-be-evaluated segment from the N to-be-evaluated segments as a highlight segment of the to-be-processed video.

[0068] In related technologies, methods for evaluating video highlights can be mainly divided into two categories: traditional evaluation methods and deep learning-based evaluation methods.

[0069] The traditional evaluation method is to judge the scene changes based on the color of the video. The richer the scene changes, the more exciting the video content.

[0070] The deep learning-based evaluation method is to extract the audio features and image features of the same time segment, merge the two into a feature vector, and then use a pre-trained model for evaluation and scoring.

[0071] Although the traditional evaluation method that relies solely on color changes to assess the excitement of video content is simple and fast, it can easily mistake scenes such as scenery as highlights, resulting in low evaluation accuracy. The difficulty of deep learning-based evaluation methods lies in the collection of training samples and the selection of training models. Poor sample collection will affect the accuracy of the evaluation method, and poor model selection will affect the speed of the evaluation method, which is not conducive to actual business use.

[0072] To address the low accuracy of assessments in related technologies, which hinders practical application, the present invention proposes a method for evaluating video highlights. This method utilizes key frames as segmentation points for highlight segments and uses texture details within the video image and inter-image motion information as the basis for evaluating the highlight level of video content. This method offers a simple and rapid assessment process, minimizes subjective influence, and offers high accuracy, making it convenient for practical application.

[0073] Specifically, the video to be processed is first divided into N segments to be evaluated, and then the motion information corresponding to each segment to be evaluated is determined based on the pixel difference between corresponding pixels between adjacent image frames in each segment to be evaluated. The texture information corresponding to each segment to be evaluated is determined based on the pixel difference between adjacent pixels in each image frame in each segment to be evaluated. The wonderful value corresponding to each segment to be evaluated is calculated based on the motion information and texture information. Then, based on the wonderful value, a parameter that represents the wonderfulness of the segment to be evaluated, at least one segment to be evaluated is determined from the N segments to be evaluated as a wonderful segment of the video to be processed.

[0074] Optionally, a larger wonderful value may indicate more wonderful video content of the segment to be evaluated.

[0075] Optionally, the wonderful values ​​corresponding to the N to-be-evaluated segments can be sorted from large to small, and the first-ranked or top-ranked to-be-evaluated segments can be selected as the wonderful segments of the video to be processed; or the to-be-evaluated segments whose wonderful values ​​are greater than or equal to a preset threshold can be selected as the wonderful segments of the video to be processed.

[0076] In the evaluation method for video highlight segments provided by the embodiment of the present invention, compared with the evaluation method based on deep learning in the related art, the evaluation method provided by the embodiment of the present invention is not limited to the collection of training samples and the selection of training models. It can calculate the highlight value corresponding to the segment to be evaluated based on the texture information and motion information corresponding to the segment to be evaluated divided from the video to be processed, and then determine at least one segment to be evaluated from the N segments to be evaluated based on the highlight value, which is a parameter that characterizes the highlight degree of the segment to be evaluated, as the highlight segment of the video to be processed. Since the calculated highlight value is not affected by subjective factors, it can objectively reflect the richness and dynamics of the video scene in the segment to be evaluated. Therefore, evaluating the segment to be evaluated based on the highlight value can effectively improve the accuracy of evaluating the highlight segments of the video.

[0077] Optionally, an embodiment of the present invention provides an implementation method for obtaining a segment to be evaluated.

[0078] The implementation of dividing the video to be processed into N segments to be evaluated may include:

[0079] Based on the time stamps of the key frames preset in the video to be processed, the video to be processed is divided into the N segments to be evaluated.

[0080] Specifically, the embodiment of the present invention uses the timestamps of the key frames pre-set in the video to be processed as the time points for segmenting the segments to be evaluated. Since the pre-set key frames can preserve the integrity of the video scenes in the video to be processed, the key frames are used to segment the video to be processed to obtain the segments to be evaluated, which can preserve the integrity of the video scenes in the segments to be evaluated, and further preserve the integrity of the video scenes in the evaluated wonderful segments.

[0081] Optionally, the implementation of segmenting the video to be processed into the N to-be-evaluated segments based on the timestamps of the key frames pre-set in the video to be processed may include:

[0082] Obtaining the timestamp of a key frame preset in the video to be processed;

[0083] Subtracting the timestamp of the first key frame from the timestamps of the key frames to obtain the timestamp of the target key frame; wherein the first key frame is a key frame whose timestamp difference with an adjacent key frame is less than a first threshold, and the adjacent key frame is a key frame adjacent to the first key frame;

[0084] Based on the timestamp of the target key frame, the video to be processed is divided into the N segments to be evaluated.

[0085] Specifically, among the preset timestamps of key frames, if the difference between the timestamps of two adjacent key frames is less than a preset first threshold, the timestamp of the latter key frame can be deleted to avoid the segmented evaluated segment being too short.

[0086] Optionally, an embodiment of the present invention provides a specific implementation method for determining motion information corresponding to each segment to be evaluated.

[0087] The method for determining the motion information corresponding to each of the segments to be evaluated based on the pixel difference between corresponding pixels between adjacent image frames in each of the segments to be evaluated may include:

[0088] Decoding each of the to-be-evaluated segments into at least two image frames;

[0089] Acquire at least two grayscale images corresponding to the at least two image frames;

[0090] Obtaining at least one first pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between corresponding pixels of adjacent grayscale images in the at least two grayscale images;

[0091] Calculate the average value mv of the pixel values ​​corresponding to all the pixel points in each of the first pixel difference maps;

[0092] An average value Mmv of mv corresponding to all first pixel difference maps in each of the segments to be evaluated is calculated as the motion information corresponding to each of the segments to be evaluated.

[0093] Specifically, each segment to be evaluated is first decoded into at least two image frames, and then the decoded image frames are grayscaled using a grayscale formula to obtain a grayscale image corresponding to the image frame;

[0094] Optionally, the grayscale formula may be the following formula (2):

[0095] gray=0.2989*R+0.5870*G+0.1140*B (2)

[0096] Among them, gray represents the pixel value of the pixel in the grayscale image, R, G, and B represent the pixel values ​​of the red, green, and blue color channels corresponding to the pixel in the image frame;

[0097] After obtaining the at least two grayscale images, at least one first pixel difference image corresponding to each to-be-evaluated segment may be obtained based on pixel differences between corresponding pixels between adjacent grayscale images in the at least two grayscale images;

[0098] Optionally, in at least two grayscale images, the absolute value of the pixel difference between corresponding pixels in adjacent grayscale images may be calculated; if the absolute value of the pixel difference is greater than or equal to a preset threshold, the pixel value of the corresponding pixel in the first pixel difference image is set to 1, otherwise it is set to 0;

[0099] For a first pixel difference map, the average value of the pixel values ​​corresponding to all pixels in the first pixel difference map can be calculated as the motion information mv of adjacent image frames. Then, for at least one first pixel difference map corresponding to each segment to be evaluated, the average value of the motion information mv of all adjacent image frames is calculated as the motion information Mmv corresponding to each segment to be evaluated.

[0100] Optionally, before obtaining the at least one first pixel difference map corresponding to each of the to-be-evaluated segments based on the pixel difference between corresponding pixels between adjacent grayscale images in the at least two grayscale images, the at least two grayscale images may be scaled proportionally to obtain at least two scaled images corresponding to the at least two grayscale images;

[0101] The method for obtaining at least one first pixel difference map corresponding to each of the to-be-evaluated segments based on the pixel difference between corresponding pixels of adjacent grayscale images in the at least two grayscale images may include:

[0102] At least one first pixel difference map corresponding to each of the to-be-evaluated segments is obtained based on pixel differences between corresponding pixel points in adjacent zoomed images in the at least two zoomed images.

[0103] Specifically, before obtaining the first pixel difference map, the grayscale image can be scaled proportionally first, and then the pixel difference between the corresponding pixel points between adjacent scaled images in the scaled image can be used to obtain at least one first pixel difference map corresponding to each segment to be evaluated. Compared with directly using the unscaled grayscale image to obtain the first pixel difference map, obtaining the first pixel difference map by using the scaled image can effectively reduce the amount of data calculation while ensuring that the image is not excessively distorted, improve the data processing speed, and thus improve the speed of evaluating the wonderful segments of the video.

[0104] For example, the grayscale image can be scaled proportionally, such as to 1 / 4 of the original size, to obtain a scaled image. The scaling method can use the area interpolation method.

[0105] It should be noted that the scaling ratio can be set based on actual conditions. In this embodiment, the scaling ratio is set to 1 / 4, which can reduce the amount of data calculation, increase the data processing speed, and ensure that the image is not excessively distorted.

[0106] After obtaining the zoom image, the difference image of two adjacent zoom images can be calculated as the first pixel difference image, that is, the absolute value of the pixel difference between the corresponding pixels of the two adjacent zoom images can be calculated. If the absolute value of the pixel difference is greater than or equal to the first preset threshold th1, the pixel value of the corresponding pixel in the first pixel difference image is set to 1, otherwise it is set to 0. The specific determination formula is as follows:

[0107]

[0108] Among them, D(i,j) represents the pixel value of the pixel point (i,j) in the difference image (first pixel difference image), A(i,j) and B(i,j) represent the pixel values ​​of the pixel point (i,j) in the two adjacent zoom images respectively, and abs() represents the absolute value function.

[0109] Optionally, an embodiment of the present invention provides a specific implementation method for determining texture information corresponding to each segment to be evaluated.

[0110] The implementation method of determining the texture information corresponding to each of the segments to be evaluated based on the pixel difference between adjacent pixels in each image frame in each of the segments to be evaluated may include:

[0111] Decoding each of the to-be-evaluated segments into at least one image frame;

[0112] Acquire at least one grayscale image corresponding to the at least one image frame;

[0113] scaling the at least one grayscale image in equal proportion to obtain at least one scaled image corresponding to the at least one grayscale image;

[0114] Obtaining at least one second pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between adjacent pixels in each of the zoomed images;

[0115] Calculate the average value tex of the pixel values ​​corresponding to all the pixel points in each of the second pixel difference images;

[0116] An average value Mtex of tex corresponding to all second pixel difference images in each of the segments to be evaluated is calculated as the texture information corresponding to each of the segments to be evaluated.

[0117] Specifically, each segment to be evaluated is first decoded into at least one image frame, and then the decoded image frame is grayscaled to obtain a grayscale image corresponding to the image frame. The grayscale formula can adopt the above formula (2).

[0118] After obtaining the grayscale image corresponding to the image frame, at least one second pixel difference image corresponding to each segment to be evaluated can be obtained based on the pixel difference between adjacent pixel points in each grayscale image; and the average value tex of the pixel values ​​corresponding to all pixel points in each second pixel difference image is calculated; then the average value Mtex of tex corresponding to all second pixel difference images in each segment to be evaluated is calculated as the texture information corresponding to each segment to be evaluated.

[0119] Optionally, the method for obtaining at least one second pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between adjacent pixels in each of the zoomed images may include:

[0120] Calculating pixel differences between adjacent pixels in a horizontal direction in each of the zoomed images to obtain at least one horizontal difference image corresponding to each of the segments to be evaluated;

[0121] Calculating pixel differences between adjacent pixels in a vertical direction in each of the zoomed images to obtain at least one vertical difference image corresponding to each of the segments to be evaluated;

[0122] At least one second pixel difference map corresponding to each of the to-be-evaluated segments is obtained based on the at least one horizontal difference map and the at least one vertical difference map.

[0123] Specifically, for the zoom image corresponding to the same grayscale image, the horizontal difference image and the vertical difference image corresponding to the zoom image can be calculated in the horizontal direction and the vertical direction respectively, and then based on the horizontal difference image and the vertical difference image corresponding to the zoom image, the second pixel difference image corresponding to the zoom image can be obtained; for at least one zoom image, at least one second pixel difference image can be obtained.

[0124] Optionally, the method of obtaining the at least one second pixel difference map corresponding to each of the to-be-evaluated segments based on the at least one horizontal difference map and the at least one vertical difference map may include:

[0125] In the horizontal difference map and the vertical difference map corresponding to each zoom map, the maximum pixel value of the corresponding pixel point is selected to form the maximum difference map corresponding to each zoom map as at least one second pixel difference map corresponding to each to-be-evaluated segment.

[0126] Alternatively, the above-mentioned scaling images can be directly used to obtain the texture information corresponding to each scaling image. The detailed process is as follows:

[0127] S1. Calculate the absolute value of the pixel difference between adjacent pixels in the horizontal direction of the zoomed image. If the absolute value of the pixel difference is greater than or equal to the second preset threshold th2, set the pixel value of the corresponding pixel in the horizontal difference image to 1; otherwise, set it to 0. Specifically, the following determination formula is used:

[0128]

[0129] Among them, V(i,j) represents the pixel value of pixel point (i,j) in the horizontal difference image, and A(i,j) and A(i,j+1) represent the pixel values ​​of adjacent pixel points (i,j) and (i,j+1) in the horizontal direction in the zoom image respectively.

[0130] S2. Calculate the absolute value of the pixel difference between adjacent pixels in the vertical direction of the zoomed image. If the absolute value of the pixel difference is greater than or equal to a second preset threshold th2, set the pixel value of the corresponding pixel in the vertical difference image to 1; otherwise, set it to 0. Specifically, the following determination formula is used:

[0131]

[0132] Among them, H(i,j) represents the pixel value of pixel point (i,j) in the vertical difference image, and A(i,j) and A(i+1,j) represent the pixel values ​​of adjacent pixel points (i,j) and (i+1,j) in the vertical direction in the zoom image respectively.

[0133] It should be noted that the preset thresholds corresponding to the acquisition of the horizontal difference map and the vertical difference map can be the same or different, and can be set according to actual conditions. In the embodiment of the present invention, the thresholds of both are set to the second preset threshold th2.

[0134] S3. Calculate the maximum pixel value of the corresponding pixel points in the horizontal difference map and the vertical difference map to obtain the maximum difference map as the second pixel difference map. The formula is as follows:

[0135] M(i,j)=max(H(i,j),V(i,j));

[0136] Among them, M(i,j) represents the pixel value of pixel point (i,j) in the maximum difference map, and max() represents the maximum value function;

[0137] Then, the average value of all pixel values ​​in the maximum difference map is calculated as the texture information tex of the corresponding image frame; and the average value Mtex of all tex obtained for each segment to be evaluated is calculated.

[0138] Optionally, the implementation of determining the excitement value corresponding to each of the to-be-evaluated segments based on the motion information and the texture information may include:

[0139] Based on the motion information Mmv and the texture information Mtex, the wonderful value corresponding to each of the segments to be evaluated is determined using formula (1):

[0140] Wonderful value = a*Mmv+(1-a)*Mtex (1)

[0141] Among them, a is the preset weight value, 0 <a<1。

[0142] Specifically, the embodiment of the present invention performs weighted averaging of texture information within the video image and motion information between the video images in the segment to be evaluated. The calculated wonderfulness value is not affected by subjective factors and can objectively reflect the richness and dynamics of the video scene.

[0143] The following example illustrates the method for evaluating video highlights provided by an embodiment of the present invention.

[0144] Figure 2 This is a second flow chart of the method for evaluating video highlights provided by the present invention. Figure 2 As shown, the detailed process of the video highlight segment evaluation method provided by the embodiment of the present invention is as follows:

[0145] 1. Input the video to be processed;

[0146] 2. Determine the segmentation time point of the video to be processed;

[0147] Specifically, the timestamps of the key frames of the video to be processed are obtained, and the timestamps of these key frames are used as the segmentation time points of the video segments (segments to be evaluated); if the difference between the timestamps of the two adjacent key frames is less than the set threshold, the latter of the two adjacent key frames is deleted.

[0148] 3. Segment the fragment to be evaluated;

[0149] Specifically, the video to be processed is segmented according to the determined segmentation time points of the segment to be evaluated;

[0150] 4. Calculate the wonderful value of the video clip;

[0151] (1) Calculate the motion information mv of the adjacent images of each segmented video segment to be evaluated;

[0152] ① Decode the fragment to be evaluated into multiple image frames;

[0153] ② Grayscale the decoded image frame to obtain a grayscale image. The grayscale conversion formula is as shown in the above formula (2);

[0154] ③ Scale the grayscale image proportionally, such as scaling it to 1 / 4 of its original size, to obtain a scaled image. The scaling method can use the regional interpolation method;

[0155] ④ Calculate the difference map (first pixel difference map) between two adjacent zoom images, that is, calculate the absolute value of the pixel difference between the corresponding pixels of the two adjacent zoom images. If the absolute value of the pixel difference is greater than or equal to the first preset threshold th1, the pixel value of the corresponding pixel in the difference map is set to 1, otherwise it is set to 0;

[0156] ⑤ Calculate the average value of all pixel values ​​in the first pixel difference map as the motion information mv of the adjacent image;

[0157] (2) Calculate the texture information tex within each segmented video image to be evaluated;

[0158] Here we calculate the texture information tex of the scaled image obtained in step ③ of (1). The detailed process is as follows:

[0159] i) calculating the absolute value of the pixel difference between adjacent pixels in the horizontal direction in the zoomed image, and if the absolute value of the pixel difference is greater than or equal to a second preset threshold th2, setting the pixel value of the corresponding pixel in the horizontal difference image to 1, otherwise setting it to 0;

[0160] ii) calculating the absolute value of the pixel difference between adjacent pixels in the zoomed image in the vertical direction. If the absolute value of the pixel difference is greater than or equal to a second preset threshold th2, the pixel value of the corresponding pixel in the vertical difference image is set to 1, otherwise it is set to 0;

[0161] iii) calculating the maximum pixel value of the corresponding pixel points in the horizontal difference map and the vertical difference map to obtain a maximum difference map (a second pixel difference map);

[0162] iv) calculating the average value of all pixel values ​​in the second pixel difference map as texture information tex in the video image;

[0163] (3) Calculate the average value Mmv of all mv obtained for each segment to be evaluated;

[0164] (4) Calculate the average value Mtex of all tex obtained for each segment to be evaluated;

[0165] (5) Calculate the wonderful value of each segment to be evaluated: wonderful value = a*Mmv+(1-a)*Mtex;

[0166] 5. Identify the highlights;

[0167] Specifically, the wonderful values ​​of all the segmented segments to be evaluated are sorted from large to small, and the segments to be evaluated that are ranked first or in the top several rankings are selected as the wonderful segments of the video to be processed.

[0168] This embodiment of the present invention uses a weighted average method that combines texture detail information within video images with motion information between video images. This method features a simple computational process, low algorithmic complexity, and high computational speed. For example, a 45-minute 1080p video can be computed in approximately two minutes on an Intel Core i7-7700HQ CPU, with high evaluation accuracy. For example, in a TV drama test, the output highlights are of varying duration, include the main characters, and feature rich, dynamic scenes.

[0169] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0170] 1. The timestamps of the key frames in the video to be processed are used as the time points for segmenting the segments to be evaluated. The duration of the resulting highlights is not fixed, and the integrity of the video scene is preserved.

[0171] 2. The texture detail information within the video image and the motion information between the video images are weighted averaged. The resulting highlight value is not affected by subjective factors and can objectively reflect the richness and dynamics of the video scene. The highlight clips are evaluated based on the richness and dynamics of the video scene. The calculation speed is fast, the accuracy is high, and the influence of subjective factors is low.

[0172] The following describes an evaluation device for video highlights provided by the present invention. The evaluation device for video highlights described below and the evaluation method for video highlights described above can refer to each other.

[0173] Figure 3 FIG. 1 is a schematic diagram of the structure of the device for evaluating video highlights provided by the present invention. Figure 3 As shown, the video highlight segment evaluation device 300 includes:

[0174] A segmentation module 301 is configured to segment the video to be processed into N segments to be evaluated, where N is an integer greater than 1;

[0175] A first determining module 302 is configured to determine motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated;

[0176] A second determining module 303 is configured to determine texture information corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each image frame in each of the segments to be evaluated;

[0177] A third determining module 304 is configured to determine a wonderful value corresponding to each of the segments to be evaluated based on the motion information and the texture information; wherein the wonderful value is used to represent the wonderfulness of each of the segments to be evaluated;

[0178] The fourth determining module 305 is configured to determine at least one segment to be evaluated from the N segments to be evaluated as a highlight segment of the video to be processed based on the highlight value corresponding to each segment to be evaluated.

[0179] In the evaluation device for video highlight segments provided by the embodiment of the present invention, compared with the evaluation method based on deep learning in the related art, the evaluation device provided by the embodiment of the present invention is not limited to the collection of training samples and the selection of training models. It can calculate the highlight value corresponding to the segment to be evaluated based on the texture information and motion information corresponding to the segment to be evaluated divided from the video to be processed, and then determine at least one segment to be evaluated from the N segments to be evaluated based on the highlight value, which is a parameter that characterizes the highlight degree of the segment to be evaluated, as the highlight segment of the video to be processed. Since the calculated highlight value is not affected by subjective factors, it can objectively reflect the richness and dynamics of the video scene in the segment to be evaluated. Therefore, evaluating the segment to be evaluated based on the highlight value can effectively improve the accuracy of evaluating the highlight segments of the video.

[0180] Optionally, the first determining module 302 is specifically configured to:

[0181] Decoding each of the to-be-evaluated segments into at least two image frames;

[0182] Acquire at least two grayscale images corresponding to the at least two image frames;

[0183] Obtaining at least one first pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between corresponding pixels of adjacent grayscale images in the at least two grayscale images;

[0184] Calculate the average value mv of the pixel values ​​corresponding to all the pixel points in each of the first pixel difference maps;

[0185] An average value Mmv of mv corresponding to all first pixel difference maps in each of the segments to be evaluated is calculated as the motion information corresponding to each of the segments to be evaluated.

[0186] Optionally, the first determining module 302 is further configured to:

[0187] Scaling the at least two grayscale images in equal proportions to obtain at least two scaled images corresponding to the at least two grayscale images;

[0188] The first determining module 302 is further specifically configured to:

[0189] At least one first pixel difference map corresponding to each of the to-be-evaluated segments is obtained based on pixel differences between corresponding pixel points in adjacent zoomed images in the at least two zoomed images.

[0190] Optionally, the second determining module 303 is specifically configured to:

[0191] Decoding each of the to-be-evaluated segments into at least one image frame;

[0192] Acquire at least one grayscale image corresponding to the at least one image frame;

[0193] scaling the at least one grayscale image in equal proportion to obtain at least one scaled image corresponding to the at least one grayscale image;

[0194] Obtaining at least one second pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between adjacent pixels in each of the zoomed images;

[0195] Calculate the average value tex of the pixel values ​​corresponding to all the pixel points in each of the second pixel difference images;

[0196] An average value Mtex of tex corresponding to all second pixel difference images in each of the segments to be evaluated is calculated as the texture information corresponding to each of the segments to be evaluated.

[0197] Optionally, the second determining module 303 is further specifically configured to:

[0198] Calculating pixel differences between adjacent pixels in a horizontal direction in each of the grayscale images to obtain at least one horizontal difference image corresponding to each of the segments to be evaluated;

[0199] Calculating pixel differences between adjacent pixels in a vertical direction in each of the grayscale images to obtain at least one vertical difference image corresponding to each of the segments to be evaluated;

[0200] At least one second pixel difference map corresponding to each of the to-be-evaluated segments is obtained based on the at least one horizontal difference map and the at least one vertical difference map.

[0201] Optionally, the second determining module 303 is further specifically configured to:

[0202] In the horizontal difference map and the vertical difference map corresponding to each zoom map, the maximum pixel value of the corresponding pixel point is selected to form the maximum difference map corresponding to each zoom map as at least one second pixel difference map corresponding to each to-be-evaluated segment.

[0203] Optionally, the segmentation module 301 is specifically configured to:

[0204] Based on the time stamps of the key frames preset in the video to be processed, the video to be processed is divided into the N segments to be evaluated.

[0205] Optionally, the segmentation module 301 is further specifically configured to:

[0206] Obtaining the timestamp of a key frame preset in the video to be processed;

[0207] Subtracting the timestamp of the first key frame from the timestamps of the key frames to obtain the timestamp of the target key frame; wherein the first key frame is a key frame whose timestamp difference with an adjacent key frame is less than a first threshold, and the adjacent key frame is a key frame adjacent to the first key frame;

[0208] Based on the timestamp of the target key frame, the video to be processed is divided into the N segments to be evaluated.

[0209] Optionally, the third determining module 304 is specifically configured to:

[0210] Based on the motion information Mmv and the texture information Mtex, the wonderful value corresponding to each of the segments to be evaluated is determined using formula (1):

[0211] Wonderful value = a*Mmv+(1-a)*Mtex (1)

[0212] Wherein, a is a preset weight value.

[0213] Figure 4 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device 400 may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the method for evaluating video highlights, which includes:

[0214] Divide the video to be processed into N segments to be evaluated, where N is an integer greater than 1;

[0215] Determining motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated;

[0216] Determining texture information corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each image frame in each of the segments to be evaluated;

[0217] Determining a wonderful value corresponding to each of the to-be-evaluated segments based on the motion information and the texture information; wherein the wonderful value is used to represent the degree of wonderfulness of each of the to-be-evaluated segments;

[0218] Based on the wonderful values ​​corresponding to the segments to be evaluated, at least one segment to be evaluated is determined from the N segments to be evaluated as a wonderful segment of the video to be processed.

[0219] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0220] In another aspect, the present invention further provides a computer program product, comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the video highlight segment evaluation method provided by the above methods, the method comprising:

[0221] Divide the video to be processed into N segments to be evaluated, where N is an integer greater than 1;

[0222] Determining motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated;

[0223] Determining texture information corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each image frame in each of the segments to be evaluated;

[0224] Determining a wonderful value corresponding to each of the to-be-evaluated segments based on the motion information and the texture information; wherein the wonderful value is used to represent the degree of wonderfulness of each of the to-be-evaluated segments;

[0225] Based on the wonderful values ​​corresponding to the segments to be evaluated, at least one segment to be evaluated is determined from the N segments to be evaluated as a wonderful segment of the video to be processed.

[0226] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for evaluating video highlights provided by the above methods is implemented, the method comprising:

[0227] Divide the video to be processed into N segments to be evaluated, where N is an integer greater than 1;

[0228] Determining motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated;

[0229] Determining texture information corresponding to each of the segments to be evaluated based on pixel differences between adjacent pixels in each image frame in each of the segments to be evaluated;

[0230] Determining a wonderful value corresponding to each of the to-be-evaluated segments based on the motion information and the texture information; wherein the wonderful value is used to represent the degree of wonderfulness of each of the to-be-evaluated segments;

[0231] Based on the wonderful values ​​corresponding to the segments to be evaluated, at least one segment to be evaluated is determined from the N segments to be evaluated as a wonderful segment of the video to be processed.

[0232] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0233] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for evaluating video highlights, characterized in that: include: Divide the video to be processed into N segments to be evaluated, where N is an integer greater than 1; Determining motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated; Decoding each of the to-be-evaluated segments into at least one image frame; Acquire at least one grayscale image corresponding to the at least one image frame; scaling the at least one grayscale image in equal proportion to obtain at least one scaled image corresponding to the at least one grayscale image; Calculating pixel differences between adjacent pixels in a horizontal direction in each of the zoomed images to obtain at least one horizontal difference image corresponding to each of the segments to be evaluated; Calculating pixel differences between adjacent pixels in a vertical direction in each of the zoomed images to obtain at least one vertical difference image corresponding to each of the segments to be evaluated; obtaining, based on the at least one horizontal difference map and the at least one vertical difference map, at least one second pixel difference map corresponding to each of the to-be-evaluated segments, wherein in the horizontal difference map and the vertical difference map, an absolute value of a pixel difference corresponding to a pixel point with a pixel value of 1 is greater than or equal to a preset threshold, and an absolute value of a pixel difference corresponding to a pixel point with a pixel value of 0 is less than the preset threshold; Calculate the average value tex of the pixel values ​​corresponding to all the pixel points in each of the second pixel difference images; Calculating an average value Mtex of tex corresponding to all second pixel difference maps in each of the segments to be evaluated as texture information corresponding to each of the segments to be evaluated; Determining a wonderful value corresponding to each of the to-be-evaluated segments based on the motion information and the texture information; wherein the wonderful value is used to represent the degree of wonderfulness of each of the to-be-evaluated segments; Based on the wonderful values ​​corresponding to the segments to be evaluated, at least one segment to be evaluated is determined from the N segments to be evaluated as a wonderful segment of the video to be processed.

2. The method for evaluating video highlights according to claim 1, wherein: The determining, based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated, motion information corresponding to each of the segments to be evaluated includes: Decoding each of the to-be-evaluated segments into at least two image frames; Acquire at least two grayscale images corresponding to the at least two image frames; Obtaining at least one first pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between corresponding pixels of adjacent grayscale images in the at least two grayscale images; Calculate the average value mv of the pixel values ​​corresponding to all the pixel points in each of the first pixel difference maps; An average value Mmv of mv corresponding to all first pixel difference maps in each of the segments to be evaluated is calculated as the motion information corresponding to each of the segments to be evaluated.

3. The method for evaluating video highlights according to claim 2, wherein: Before obtaining at least one first pixel difference map corresponding to each of the to-be-evaluated segments based on pixel differences between corresponding pixels in adjacent grayscale images of the at least two grayscale images, the method further includes: Scaling the at least two grayscale images in equal proportions to obtain at least two scaled images corresponding to the at least two grayscale images; The obtaining, based on pixel differences between corresponding pixels between adjacent grayscale images in the at least two grayscale images, at least one first pixel difference image corresponding to each of the to-be-evaluated segments comprises: At least one first pixel difference map corresponding to each of the to-be-evaluated segments is obtained based on pixel differences between corresponding pixel points in adjacent zoomed images in the at least two zoomed images.

4. The method for evaluating video highlights according to claim 1, wherein: The obtaining, based on the at least one horizontal difference map and the at least one vertical difference map, at least one second pixel difference map corresponding to each of the to-be-evaluated segments includes: In the horizontal difference map and the vertical difference map corresponding to each zoom map, the maximum pixel value of the corresponding pixel point is selected to form the maximum difference map corresponding to each zoom map as at least one second pixel difference map corresponding to each to-be-evaluated segment.

5. The method for evaluating video highlights according to claim 1, wherein: The step of dividing the video to be processed into N segments to be evaluated includes: Based on the time stamps of the key frames preset in the video to be processed, the video to be processed is divided into the N segments to be evaluated.

6. The method for evaluating video highlights according to claim 5, wherein: The step of dividing the video to be processed into the N segments to be evaluated based on the timestamps of the key frames preset in the video to be processed includes: Obtaining the timestamp of a key frame preset in the video to be processed; Subtracting the timestamp of the first key frame from the timestamps of the key frames to obtain the timestamp of the target key frame; wherein the first key frame is a key frame whose timestamp difference with an adjacent key frame is less than a first threshold, and the adjacent key frame is a key frame adjacent to the first key frame; Based on the timestamp of the target key frame, the video to be processed is divided into the N segments to be evaluated.

7. The method for evaluating video highlights according to claim 1, wherein: The determining, based on the motion information and the texture information, a wonderful value corresponding to each of the to-be-evaluated segments includes: Based on the motion information Mmv and the texture information Mtex, the wonderful value corresponding to each of the segments to be evaluated is determined using formula (1): Wonderful value = a*Mmv+(1-a)*Mtex (1) where a is a preset weight value.

8. A device for evaluating video highlights, characterized in that: include: A segmentation module is used to segment the video to be processed into N segments to be evaluated, where N is an integer greater than 1; A first determining module is configured to determine motion information corresponding to each of the segments to be evaluated based on pixel differences between corresponding pixels between adjacent image frames in each of the segments to be evaluated; A second determining module is configured to decode each of the segments to be evaluated into at least one image frame; and for obtaining at least one grayscale image corresponding to the at least one image frame; and performing proportional scaling on the at least one grayscale image to obtain at least one scaled image corresponding to the at least one grayscale image; and calculating pixel differences between adjacent pixels in the horizontal direction in each of the zoomed images to obtain at least one horizontal difference image corresponding to each of the segments to be evaluated; and calculating pixel differences between adjacent pixels in the vertical direction in each of the zoomed images to obtain at least one vertical difference map corresponding to each of the segments to be evaluated; and obtaining at least one second pixel difference map corresponding to each of the segments to be evaluated based on the at least one horizontal difference map and the at least one vertical difference map, wherein the absolute value of the pixel difference corresponding to the pixel point with a pixel value of 1 in the horizontal difference map and the vertical difference map is greater than or equal to a preset threshold, and the absolute value of the pixel difference corresponding to the pixel point with a pixel value of 0 is less than the preset threshold; and for calculating an average value tex of pixel values ​​corresponding to all pixel points in each of the second pixel difference maps; and calculating an average value Mtex of tex corresponding to all second pixel difference maps in each of the segments to be evaluated as texture information corresponding to each of the segments to be evaluated; A third determining module is configured to determine a wonderful value corresponding to each of the segments to be evaluated based on the motion information and the texture information; wherein the wonderful value is used to represent the wonderfulness of each of the segments to be evaluated; The fourth determining module is configured to determine, based on the highlight values ​​corresponding to the respective to-be-evaluated segments, at least one to-be-evaluated segment from the N to-be-evaluated segments as a highlight segment of the to-be-processed video.

Images