Video segment scoring method and apparatus, electronic device, and medium

Abstract

The application discloses a video segment scoring method and device, electronic equipment and medium, and belongs to the technical field of video processing. The method comprises the following steps: obtaining one or more first video segments of a video; generating a first feature map of each first video segment; updating the first feature map of each first video segment based on a second feature map corresponding to each first video segment to obtain a target feature map of each first video segment, wherein the second feature map corresponding to the first video segment is used to represent the distinguishing degree of the video segment before the first video segment in the feature; and obtaining a first score of each first video segment based on the target feature map of each first video segment to obtain one or more first scores of one or more first video segments.

Description

Technical Field

[0001] This application belongs to the field of video processing technology, specifically relating to a method, apparatus, electronic device, and medium for scoring video clips. Background Technology

[0002] Today, video-enabled devices such as smartphones and computers can automatically edit videos stored on the device and help users automatically filter out highlight segments. However, current video highlight selection algorithms primarily rely on scoring video segments based solely on their inherent features and selecting segments as highlights according to their scores. This scoring method often fails to identify true highlight segments, meaning its accuracy is typically poor. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, electronic device, and medium for scoring video clips, which can solve the technical problem of poor accuracy in the first scoring of existing video clips.

[0004] In a first aspect, embodiments of this application provide a method for scoring video clips, the method comprising:

[0005] Retrieve one or more first video segments from the video;

[0006] Generate a first feature map for each of the first video segments;

[0007] Based on the second feature map corresponding to each of the first video segments, the first feature map of each of the first video segments is updated to obtain the target feature map of each of the first video segments. The second feature map corresponding to the first video segment is used to characterize the feature distinguishability of the video segments preceding the first video segment.

[0008] Based on the target feature map of each of the first video segments, a first score is obtained for each of the first video segments, resulting in one or more first scores for the one or more first video segments.

[0009] Secondly, embodiments of this application provide a video segment scoring device, the device comprising:

[0010] The acquisition module is used to acquire one or more first video segments from the video.

[0011] A generation module is used to generate a first feature map for each of the first video segments;

[0012] The update module is used to update the first feature map of each first video segment based on the second feature map corresponding to each first video segment to obtain the target feature map of each first video segment, wherein the second feature map corresponding to the first video segment is used to characterize the feature discrimination of the video segments preceding the first video segment.

[0013] The scoring module is used to obtain a first score for each of the first video segments based on the target feature maps of each of the first video segments, thereby obtaining one or more first scores for the one or more first video segments.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method provided in the first aspect.

[0015] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method provided in the first aspect.

[0016] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the method provided in the first aspect.

[0017] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as provided in the first aspect.

[0018] In the video segment scoring method, apparatus, electronic device, and medium of this application, one or more first video segments can be acquired from a video. During the scoring process of the first video segments, a first feature map of the first video segment is first generated to represent its video features. Then, the first feature map is updated using a second feature map corresponding to the first video segment to obtain a target feature map. Finally, the first video segment is scored based on the target feature map. Since the second feature map reflects the differences in features between video segments preceding the first video segment, the target feature map can reflect both the features of the first video segment and the distinguishing features between the first video segment and its preceding segments. Therefore, a more accurate first score can be obtained, thereby improving the accuracy of the scoring. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a video clip scoring method provided in one embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating a video segment scoring method provided in another embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of a video clip scoring device provided in another embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] To address the aforementioned technical problems, this application provides a method for scoring video segments. The scoring method for video segments provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0027] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a video segment scoring method provided in one embodiment of this application. This embodiment of the application provides a video segment scoring method, which may include:

[0028] S101, Obtain one or more first video segments from the video;

[0029] In this embodiment, the scoring of video segments is performed by a terminal device, which can be a mobile phone or a computer. The video can be captured by the application terminal device or stored on the terminal device. The first video segment is a segment extracted from the video. After extracting more than one first video segment from the video, the duration of each first video segment can be the same, and N first video frames can be obtained from each first video segment.

[0030] For example, N is 4. The first video segment can be a 2-second video segment, with a 0.5-second interval between two adjacent first video segments. Four first video frames can be obtained from each first video segment, and these four video frames are used to characterize the features of the first video segment. A sliding window with a fixed duration of 2 seconds can be set to extract the first video segment starting from the beginning of the video. The sliding window step size is 0.5 seconds, and one first video segment is extracted each time the window slides one step.

[0031] In addition, four frames can be extracted at equal intervals from the first video segment. For example, a first video frame can be extracted at the 0th, 40th, 80th, and 120th seconds of the first video segment.

[0032] S102, Generate a first feature map for each of the first video segments;

[0033] In this embodiment, for each first video segment, video features of the first video segment can be extracted from the first video segment to generate a first feature map.

[0034] As an optional embodiment, N first video frames can be extracted from the first video segment, and then the N first video frames can be superimposed in chronological order within the first video segment. The superimposed N first video frames are then input into an encoding network to convert them into a first feature map of each first video frame, which serves as the first feature map of the first video segment.

[0035] The first feature map is a representation of the image features of the first video frame extracted at different levels of the convolutional neural network. It can reflect the image features of the corresponding first video frame and further reflect the video features of the first video segment. For example, the first feature map can reflect the edge features, color features, and texture features of the first video segment, etc.

[0036] S103, based on the second feature map corresponding to each of the first video segments, update the first feature map of each of the first video segments to obtain the target feature map of each of the first video segments, wherein the second feature map corresponding to the first video segment is used to characterize the feature distinguishability of the video segments preceding the first video segment.

[0037] In this embodiment, the second feature map corresponding to the first video segment is a feature map reflecting the features of video segments preceding the first video segment. The first feature map can be adjusted using the second feature map to obtain the target feature map of the first video segment. Therefore, the target feature map not only reflects the features of the first video segment but also reflects the differences in features between the first video segment and the video segments preceding it.

[0038] As an optional embodiment, the number of video frames obtainable in the video segments preceding the first video segment is an integer P. If P is greater than or equal to N, where N is the number of the first video frames, then N second feature maps can be determined based on the P video frames, and the N first feature maps corresponding to the first video segment can be updated to obtain N target feature maps. If P is less than N, then P second feature maps can be determined based on the P video frames, and (NP) all-zero feature maps can be added as second feature maps, where the all-zero feature map is a feature map where all feature values ​​of all pixels are 0.

[0039] S104, based on the target feature map of each of the first video segments, obtain the first score of each of the first video segments, and obtain one or more first scores of the one or more first video segments.

[0040] In this embodiment of the application, for each first video segment, after obtaining the target feature map of the first video segment, the target feature map can be input into a pre-trained scoring model, and the scoring model can then output the first score corresponding to the first video segment. The first score refers to an indicator that quantitatively evaluates the visual appeal or salience of the first video segment.

[0041] For example, a scoring model can consist of more than one consecutive fully connected layer. In this layered structure, earlier layers might capture simple features such as edges and textures. As the layers deepen, the fully connected layers capture more abstract features, such as object parts and shapes. The final fully connected layer combines these abstract features to generate and output the final first score.

[0042] In this application, one or more first video segments can be obtained from a video. During the scoring process of the first video segments, a first feature map of the first video segment is first generated to represent its video features. Then, the first feature map is updated using a second feature map corresponding to the first video segment to obtain a target feature map. Finally, the first video segment is scored based on the target feature map. Since the second feature map reflects the feature differences between video segments preceding the first video segment, the target feature map can reflect both the features of the first video segment and the distinguishing features between the first video segment and its preceding segments. Therefore, a more accurate first score can be obtained, thereby improving the accuracy of the scoring.

[0043] In some embodiments, after S104, the method further includes:

[0044] Highlight segments are determined from the video based on one or more of the first scores;

[0045] The highlight segment is shown.

[0046] In this embodiment, after determining the first score of each video, it is also necessary to consider the combined video segments obtained when more than one consecutive first video segments are combined. The first video segments and the combined video segments can be collectively referred to as second video segments, and the second score of each second video segment is uniformly obtained. Highlight segments are determined from the second video segments based on the level of the second score.

[0047] In addition, after the highlight segment is determined, it can be displayed on the terminal device's display page. Users can edit the highlight segment on the display page to crop it from the selected highlight segment, obtain the target segment that the user wants to save, and store the target segment.

[0048] For example, highlight clips can be displayed on a terminal device in the form of a timeline, with each position on the timeline corresponding to a point in time within the highlight clip. Users can manually select the start and end points within the highlight clip and save the video segment between the start and end points as the stored target segment.

[0049] In this embodiment, highlight segments can be selected based on a first score and displayed, which can intuitively show users the automatically captured highlight segments and facilitate further processing of the highlight segments.

[0050] In some embodiments, determining the highlight segment from the video based on the one or more first scores includes:

[0051] One or more scaling factors are determined based on the one or more first scores, wherein each scaling factor is used to characterize the scaling ratio of the first score corresponding to the first number of first video segments included in the highlight segment;

[0052] Based on the one or more first scores, one or more range scores are determined, wherein each range score is a combined score corresponding to a second video segment obtained by combining at least one consecutive first video segment;

[0053] A second score is determined for one or more second video segments based on the one or more range scores and the one or more scaling factors;

[0054] The highlight segments of the video are determined based on the second score in one or more second video segments.

[0055] In this embodiment, after determining the first score of one or more first video segments, it is necessary to classify the first video segments and combined video segments into second video segments for comparison. Since the duration of each second video segment is different, it is necessary to determine the range score of each second video segment based on one or more first scores, and determine different scaling factors according to the different number of first video segments included in the second video segment. The second score of each second video segment is obtained based on the range score and scaling factor of each second video segment, and the highlight segments are determined from the second video segments based on the level of the second score.

[0056] For example, after obtaining the second score for each second video segment, the second video segment with the highest second score can be identified as the highlight segment. Alternatively, second video segments can be randomly selected from all second video segments with a second score greater than a score threshold.

[0057] By using the above method, a unified scoring standard can be established for video segments of different durations, making it easier to accurately select highlight segments from the video.

[0058] In some embodiments, determining one or more scaling factors based on the one or more first scores includes:

[0059] Determine the average score and the maximum score of the above or more first ratings;

[0060] Obtain the maximum and minimum duration of the highlight fragment;

[0061] Obtain the step duration difference between two adjacent first video segments;

[0062] The one or more scaling factors are determined based on the average score, the maximum score, the maximum sustainable duration, the minimum sustainable duration, and the step duration.

[0063] In this embodiment, the average of one or more first scores, i.e., the score mean, can be calculated. The maximum value can also be selected from one or more first scores, i.e., the score maximum. Furthermore, the maximum and minimum durations of the highlight segments can be obtained. The maximum duration refers to the maximum achievable duration of the highlight segment, and the minimum duration refers to the minimum achievable duration of the highlight segment. The maximum and minimum durations can be preset by those skilled in the art or by the user of the terminal device. Additionally, the step duration between any two adjacent first video segments needs to be obtained, i.e., the time interval or span between two adjacent first video segments. Then, the aforementioned score mean, score maximum, maximum duration, minimum duration, and step duration can be used to calculate the scaling factor corresponding to the first number of first video segments included in each second video segment.

[0064] As an optional embodiment, the scaling factor can be calculated using a scaling factor formula, as follows:

[0065]

[0066] Where i = k-1, k is the first quantity, S mean S represents the average score. max d represents the maximum score. max For the maximum sustainable duration, d min The minimum sustainable duration, dur is the step duration, D0 is the scaling factor when the first quantity is 1, and D i This is the scaling factor when the first quantity is (i+1).

[0067] Using the above method, the scaling factor for video segments of different durations can be accurately calculated.

[0068] In some embodiments, determining one or more range scores based on the one or more first scores includes:

[0069] For the third video segment, determine the fourth video segment in the third video segment, wherein the third video segment is any second video segment, and the fourth video segment is the last first video segment in the video segment ordered by time.

[0070] Determine the first sequence number of the fourth video segment in the video;

[0071] Based on the first sequence number and at least one first score corresponding to at least one first video segment in the third video segment, the range score of the third video segment is determined.

[0072] In this embodiment, before calculating the range score of each second video segment, the first video segments in the video can be sorted chronologically. Then, for the third video segment, after knowing the first sequence number of the fourth video segment in the third video segment and the first number of first video segments included in the third video segment, it is possible to determine which first video segments are included in the third video segment, and determine the range score of the third video segment based on at least one first score corresponding to these first video segments.

[0073] For example, if the first sequence number of the fourth video segment in the third video segment is 7, and the first number of the first video segments included in the third video segment is 4, then the third video segment includes the first video segments with sequence numbers 4-7, and the range score of the third video segment can be determined based on the 4 first scores corresponding to the first video segments with sequence numbers 4-7.

[0074] For example, the range fraction equation can be used to determine the range fraction of each third video segment. The range fraction equation is as follows:

[0075]

[0076] Where i = k-1, k is the first quantity, j = q-1, q is the first sequence number, n is the sequence number of any first video segment, and S n X represents the first rating corresponding to the first video segment with index n. (i,j) This represents the range score of the second video segment whose first sequence number is p and includes k first video segments.

[0077] In this embodiment, the range scores of video segments with different durations can be accurately calculated.

[0078] In some embodiments, determining a second rating for one or more second video segments based on the one or more range scores and the one or more scaling factors includes:

[0079] Query the second number of the first video segment in each second video segment;

[0080] Determine the first scaling factor corresponding to the second quantity;

[0081] The product of the range score corresponding to the second video segment and the first scaling factor is determined as the second score of the second video segment.

[0082] In this embodiment, after calculating the scaling factor and the range score, one or more scaling factors and one or more range scores can be obtained. For each second video segment, a second number of first video segments included in the second video segment can be determined, and the first scaling factor corresponding to the second number can be queried. Then, the product of the range score corresponding to the second video segment and the first scaling factor is determined as the second score of the second video segment.

[0083] For example, the final score for each second video segment can be calculated using a second scoring formula, where the second scoring formula is:

[0084] Y (i,j) =X (i,j) ×D i

[0085] Where i = k-1, k is the first quantity, j = q-1, q is the first sequence number, X (i,j) D represents the range score of the second video segment whose first index is p and includes k first video segments. i Y is the scaling factor when the first quantity is (i+1). (i,j) This represents the second score of the second video segment, which has a first sequence number of p and includes k first video segments.

[0086] Based on the second scoring formula described above, as follows Figure 2 As shown, assuming the highlight sequence is [1,2,3,4], the first score of the first video segment is [1,2,3,2]. The scaling factors for the first number of segments (1,2,3,4) are calculated to be 0.5, 0.75, 0.9, and 1, respectively. A scaling factor matrix is ​​then set, and the range score of each second video segment can be calculated to obtain the range score matrix. Finally, multiplying the range score of each second video segment by its corresponding scaling factor yields the second score of that second video segment, generating the second score matrix.

[0087] For example, the second highest-rated video clip was the video clip of the continuous sequence 2-4, with a score of 2.07; the second choice was the clip of the entire sequence 1-4, with a score of 2; and the third was the second video clip of the continuous sequence 2-3, with a score of 1.88.

[0088] Based on this embodiment, the second score of video segments with different durations is accurately calculated, which facilitates the selection of highlight segments.

[0089] In some embodiments, updating the first feature map of each first video segment based on the second feature map corresponding to each first video segment to obtain the target feature map of each first video segment includes:

[0090] For each of the first video segments, the first feature map of the first video segment and the second feature map corresponding to the first video segment are superimposed and fused to obtain a fused feature map.

[0091] The fused feature map is split to obtain the target feature map.

[0092] In this embodiment, for each first video segment, the channels of its first feature map and the corresponding second feature map can be merged together to obtain a new superimposed feature map. For example, if the first feature map has 128 channels and the second feature map also has 128 channels, then the superimposed feature map will have 256 channels. The purpose of this is to fuse feature information from two different levels together to produce a richer feature representation.

[0093] The overlay feature map includes not only the segment features associated with the first video segment, but also the feature segments preceding the first video segment and the discriminative features of the first video segment. The discriminative features and segment features can be fused using a trained fusion model to obtain a fused feature map. The fused feature map performs feature fusion on the segment features and discriminative features, thus providing a more comprehensive representation of the features of the first video segment.

[0094] After obtaining the fused feature map, it can be split according to the pre-set rules to obtain the third feature map and the fourth feature map. The third feature map can correspond to the features of the first feature map, but the feature values ​​of each pixel in the third feature map are adjusted based on the second feature map, which can better represent the features of the first video segment. The third feature map can be the target feature map.

[0095] The fourth feature map can correspond to the features of the second feature map, but the feature values ​​of each pixel in the fourth feature map have been adjusted based on the first feature map. The fourth feature map can be used as the second feature map corresponding to the video segments in the subsequent video segments of the first video segment.

[0096] By using the above method, the target feature map of the first video segment can not only reflect the features of the first video segment, but also reflect the distinguishing features between the first video segment and the feature segments preceding it, thus providing a richer representation of the features of the first video segment.

[0097] In addition, as an optional embodiment, the target feature map can be input into a pre-trained scoring model, which can then output the first score corresponding to the first video segment corresponding to the target feature map.

[0098] The scoring model can be scored using collected training videos. The training process involves dividing the training video into multiple training video segments at intervals of a first duration, manually assigning a score to each segment as its true highlight value. Then, N training video frames are extracted at equal intervals from each segment. These N frames are sequentially stacked and converted into a fifth feature map, which is the feature map corresponding to that training video segment.

[0099] Similarly, the fifth feature map is adjusted using the sixth feature map, which represents the features of the previous video segments, to obtain the seventh feature map. The seventh feature map is then input into the scoring model to obtain the model highlight value output by the scoring model.

[0100] Then, the model's highlight values ​​are compared with the real highlight values, which allows for the adjustment of model parameters in the scoring model to train the scoring model.

[0101] Figure 3 This is a schematic diagram of the structure of a video clip scoring device provided in another embodiment of this application, as shown below. Figure 3 As shown, the scoring device for this video clip may include:

[0102] Acquisition module 301 is used to acquire one or more first video segments from the video;

[0103] Generation module 302 is used to generate a first feature map for each of the first video segments;

[0104] The update module 303 is used to update the first feature map of each first video segment based on the second feature map corresponding to each first video segment to obtain the target feature map of each first video segment, wherein the second feature map corresponding to the first video segment is used to characterize the feature distinguishability of the video segments preceding the first video segment.

[0105] The scoring module 304 is used to obtain a first score for each of the first video segments based on the target feature maps of each of the first video segments, thereby obtaining one or more first scores for the one or more first video segments.

[0106] In this application, one or more first video segments can be obtained from a video. During the scoring process of the first video segments, a first feature map of the first video segment is first generated to represent its video features. Then, the first feature map is updated using a second feature map corresponding to the first video segment to obtain a target feature map. Finally, the first video segment is scored based on the target feature map. Since the second feature map reflects the feature differences between video segments preceding the first video segment, the target feature map can reflect both the features of the first video segment and the distinguishing features between the first video segment and its preceding segments. Therefore, a more accurate first score can be obtained, thereby improving the accuracy of the scoring.

[0107] In another alternative example, the scoring device for the video segment further includes:

[0108] A determining module is used to determine highlight segments from the video based on one or more first scores;

[0109] The display module is used to display the highlight fragment.

[0110] In another alternative example, the determining module may also include:

[0111] The first determining unit is configured to determine one or more scaling factors based on the one or more first scores, wherein each scaling factor is used to characterize the scaling ratio of the first score corresponding to the first number of first video segments included in the highlight segment;

[0112] The second determining unit is used to determine one or more range scores based on the one or more first scores, wherein each range score is a combined score corresponding to a second video segment obtained by combining at least one consecutive first video segment;

[0113] A scoring unit is used to determine a second score for one or more second video segments based on the one or more range scores and the one or more scaling factors;

[0114] The third determining unit is used to determine the highlight segment of the video in the one or more second video segments based on the second score.

[0115] In another alternative example, the first determining unit may also be used to:

[0116] Determine the average score and the maximum score of the one or more first ratings;

[0117] Obtain the maximum and minimum duration of the highlight fragment;

[0118] Obtain the step duration difference between two adjacent first video segments;

[0119] The one or more scaling factors are determined based on the average score, the maximum score, the maximum sustainable duration, the minimum sustainable duration, and the step duration.

[0120] In another alternative example, the second determining unit may also be used for:

[0121] For the third video segment, determine the fourth video segment in the third video segment, wherein the third video segment is any second video segment, and the fourth video segment is the last first video segment in the video segment ordered by time.

[0122] Determine the first sequence number of the fourth video segment in the video;

[0123] Based on the first sequence number and at least one first score corresponding to at least one first video segment in the third video segment, the range score of the third video segment is determined.

[0124] In another alternative example, the scoring unit can also be used for:

[0125] Query the second number of the first video segment in each second video segment;

[0126] Determine the first scaling factor corresponding to the second quantity;

[0127] The product of the range score corresponding to the second video segment and the first scaling factor is determined as the second score of the second video segment.

[0128] In another optional example, the processor 510 is further configured to perform channel superposition and fusion of the first feature map of the first video segment and the second feature map corresponding to the first video segment for each of the first video segments to obtain a fused feature map;

[0129] The fused feature map is split to obtain the target feature map.

[0130] The video segment scoring device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device to these types of devices.

[0131] The video segment scoring device in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit the specific operating system used.

[0132] The video segment scoring device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0133] Optionally, such as Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 410, a memory 409, and a program or instructions stored in the memory 409 and executable on the processor 410. When the program or instructions are executed by the processor 410, they implement the various processes of the above-described video segment scoring method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0134] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0135] Please refer to the following: Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 500 includes, but is not limited to, components such as: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, and a processor 510.

[0136] Those skilled in the art will understand that the electronic device 500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0137] The processor 510 is used to acquire one or more first video segments from the video.

[0138] Processor 510 is configured to generate a first feature map for each of the first video segments;

[0139] The processor 510 is configured to update the first feature map of each first video segment based on the second feature map corresponding to each first video segment to obtain the target feature map of each first video segment, wherein the second feature map corresponding to the first video segment is used to characterize the feature-discriminativeness of video segments preceding the first video segment.

[0140] Processor 510 is configured to perform a first score on each of the first video segments based on the target feature maps of each of the first video segments, thereby obtaining one or more first scores for the one or more first video segments.

[0141] In this application, one or more first video segments can be obtained from a video. During the scoring process of the first video segments, a first feature map of the first video segment is first generated to represent its video features. Then, the first feature map is updated using a second feature map corresponding to the first video segment to obtain a target feature map. Finally, the first video segment is scored based on the target feature map. Since the second feature map reflects the feature differences between video segments preceding the first video segment, the target feature map can reflect both the features of the first video segment and the distinguishing features between the first video segment and its preceding segments. Therefore, a more accurate first score can be obtained, thereby improving the accuracy of the scoring.

[0142] In another alternative example, the scoring device for the video segment further includes:

[0143] Processor 510 is configured to determine highlight segments from the video based on one or more first scores;

[0144] Display unit 506 is used to display the highlight segment.

[0145] In another alternative example, processor 510 may also include:

[0146] The first determining unit is configured to determine one or more scaling factors based on the one or more first scores, wherein each scaling factor is used to characterize the scaling ratio of the first score corresponding to the first number of first video segments included in the highlight segment;

[0147] The second determining unit is used to determine one or more range scores based on the one or more first scores, wherein each range score is a combined score corresponding to a second video segment obtained by combining at least one consecutive first video segment;

[0148] A scoring unit is used to determine a second score for one or more second video segments based on the one or more range scores and the one or more scaling factors;

[0149] The third determining unit is used to determine the highlight segment of the video in the one or more second video segments based on the second score.

[0150] In another alternative example, the first determining unit may also be used to:

[0151] Determine the average score and the maximum score of the one or more first ratings;

[0152] Obtain the maximum and minimum duration of the highlight fragment;

[0153] Obtain the step duration difference between two adjacent first video segments;

[0154] The one or more scaling factors are determined based on the average score, the maximum score, the maximum sustainable duration, the minimum sustainable duration, and the step duration.

[0155] In another alternative example, the second determining unit may also be used for:

[0156] For the third video segment, determine the fourth video segment in the third video segment, wherein the third video segment is any second video segment, and the fourth video segment is the last first video segment in the video segment ordered by time.

[0157] Determine the first sequence number of the fourth video segment in the video;

[0158] Based on the first sequence number and at least one first score corresponding to at least one first video segment in the third video segment, the range score of the third video segment is determined.

[0159] In another alternative example, the scoring unit can also be used for:

[0160] Query the second number of the first video segment in each second video segment;

[0161] Determine the first scaling factor corresponding to the second quantity;

[0162] The product of the range score corresponding to the second video segment and the first scaling factor is determined as the second score of the second video segment.

[0163] In another optional example, the processor 510 is further configured to perform channel superposition and fusion of the first feature map of the first video segment and the second feature map corresponding to the first video segment for each of the first video segments to obtain a fused feature map;

[0164] The fused feature map is split to obtain the target feature map.

[0165] It should be understood that, in this embodiment, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0166] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 509 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0167] Processor 510 may include one or more processing units; optionally, processor 510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 510.

[0168] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video segment scoring method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0169] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0170] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described video segment scoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0171] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0172] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video clip scoring method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0173] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0175] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for scoring video clips, characterized in that, include: Retrieve one or more first video segments from the video; Generate a first feature map for each of the first video segments; Based on the second feature map corresponding to each of the first video segments, the first feature map of each of the first video segments is updated to obtain the target feature map of each of the first video segments. The second feature map corresponding to the first video segment is used to characterize the feature distinguishability of the video segments preceding the first video segment. Based on the target feature map of each first video segment, a first score is obtained for each first video segment, and one or more first scores are obtained for the one or more first video segments. One or more scaling factors are determined based on the one or more first scores, wherein each scaling factor is used to characterize the scaling ratio of the first score corresponding to the first number of first video segments included in the highlight segment; Based on the one or more first scores, one or more range scores are determined, wherein each range score is a combined score corresponding to a second video segment obtained by combining at least one consecutive first video segment; A second score is determined for one or more second video segments based on the one or more range scores and the one or more scaling factors; Based on the second score, the highlight segments of the video are determined in one or more second video segments; Determining one or more scaling factors based on one or more first scores includes: Determine the average score and the maximum score of the above or more first ratings; Obtain the maximum and minimum duration of the highlight fragment; Obtain the step duration difference between two adjacent first video segments; The one or more scaling factors are determined based on the average score, the maximum score, the maximum sustainable duration, the minimum sustainable duration, and the step duration. The determination of one or more score ranges based on one or more first scores includes: For the third video segment, determine the fourth video segment in the third video segment, wherein the third video segment is any second video segment, and the fourth video segment is the last first video segment in the video segment ordered by time. Determine the first sequence number of the fourth video segment in the video; Based on the first sequence number and at least one first score corresponding to at least one first video segment in the third video segment, the range score of the third video segment is determined.

2. The method according to claim 1, characterized in that, After determining the highlight segments of the video based on the second score in the one or more second video segments, the method further includes: The highlight segment is shown.

3. The method according to claim 1, characterized in that, The second rating for determining one or more second video segments based on the one or more range scores and the one or more scaling factors includes: Query the second number of the first video segment in each second video segment; Determine the first scaling factor corresponding to the second quantity; The product of the range score corresponding to the second video segment and the first scaling factor is determined as the second score of the second video segment.

4. The method according to claim 1, characterized in that, The step of updating the first feature map of each first video segment based on the second feature map corresponding to each first video segment to obtain the target feature map of each first video segment includes: For each of the first video segments, the first feature map of the first video segment and the second feature map corresponding to the first video segment are superimposed and fused to obtain a fused feature map. The fused feature map is split to obtain the target feature map.

5. A video clip scoring device, characterized in that, include: The acquisition module is used to acquire one or more first video segments from the video. A generation module is used to generate a first feature map for each of the first video segments; The update module is used to update the first feature map of each first video segment based on the second feature map corresponding to each first video segment to obtain the target feature map of each first video segment, wherein the second feature map corresponding to the first video segment is used to characterize the feature discrimination of the video segments preceding the first video segment. The scoring module is used to obtain a first score for each of the first video segments based on the target feature maps of each of the first video segments, thereby obtaining one or more first scores for the one or more first video segments. A determining module is configured to: determine one or more scaling factors based on the one or more first scores, wherein each scaling factor is used to characterize the scaling ratio of the first score corresponding to a first number of first video segments included in the highlight segment; determine one or more range scores based on the one or more first scores, wherein each range score is a combined score corresponding to a second video segment obtained by combining at least one consecutive first video segment; determine a second score for one or more second video segments based on the one or more range scores and the one or more scaling factors; and determine the highlight segment of the video in the one or more second video segments based on the second score. The determining module is specifically used for: Determine the average score and the maximum score of the above or more first ratings; Obtain the maximum and minimum duration of the highlight fragment; Obtain the step duration difference between two adjacent first video segments; The one or more scaling factors are determined based on the average score, the maximum score, the maximum sustainable duration, the minimum sustainable duration, and the step duration. The determining module is specifically used for: For the third video segment, determine the fourth video segment in the third video segment, wherein the third video segment is any second video segment, and the fourth video segment is the last first video segment in the video segment ordered by time. Determine the first sequence number of the fourth video segment in the video; Based on the first sequence number and at least one first score corresponding to at least one first video segment in the third video segment, the range score of the third video segment is determined.

6. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the scoring method for the video segment as described in any one of claims 1-4.

7. A readable storage medium, characterized in that, A program or instruction is stored on a readable storage medium, which, when executed by a processor, implements the steps of the scoring method for a video segment as described in any one of claims 1-4.

Images