A video summarization method and apparatus

Abstract

The present disclosure provides a video summary generation method and device, wherein the method comprises: segmenting a target subject from a first video frame of a video; segmenting each segmentation graph of the target subject in time sequence from each second video frame of the video; the second video frame is a video frame after the first video frame in the video; and merging the target subject and the segmentation graph of the target subject in time sequence with a background of the video in time sequence to obtain a video summary of the video. The embodiment can improve the accuracy and applicability of generating the video summary, and improve the fineness of the video summary.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a video summarization method and apparatus. Background Technology

[0002] Video summarization refers to compressing a longer video into a shorter one, while retaining as much of the original video's content as possible. Current technologies typically rely on foreground detection and background modeling to generate video summaries. However, when using small targets as the foreground, this can lead to missed foreground elements, reducing the accuracy of the generated video summaries. Furthermore, while some existing technologies use object detection algorithms to detect targets in the original video and then generate summaries, these cannot detect subjects outside of predefined target groups, lacking in both precision and applicability. Summary of the Invention

[0003] In view of this, the present disclosure provides at least one video summarization method, apparatus, electronic device, and storage medium, which can improve the applicability of generating accurate video summaries and enhance the precision of video summaries.

[0004] In a first aspect, embodiments of this disclosure provide a video summary generation method, including:

[0005] Segment the target subject from the first video frame;

[0006] The target subject is segmented temporally from each of the second video frames; the second video frame is the video frame following the first video frame.

[0007] The target subject and its temporal segmentation map are merged with the video background in chronological order to obtain the video summary.

[0008] Optionally, segmentation maps of the target subject in time are extracted from each of the second video frames of the video, including:

[0009] Identify the subject image in the second video frame that includes the target subject;

[0010] The main image corresponding to the second video frame is segmented to generate a mask image corresponding to the main image; wherein, the main image and the mask image have the same size; the pixel value of each pixel in the mask image indicates whether each pixel in the mask image belongs to the target main body;

[0011] Based on the position information of the subject image corresponding to the second video frame, the mask image is mapped onto the second video frame to determine the segmentation map of the target subject on the second video frame;

[0012] The segmentation maps of the target subject in each video frame are sorted according to the temporal sequence of each video frame to obtain the segmentation maps of the target subject in the temporal sequence.

[0013] Optionally, the target subject segmented from the first video frame may be multiple; before merging the target subject and its temporal segmentation map with the video background in chronological order, the method further includes:

[0014] In response to the overlap of segmentation maps of different target subjects on any video frame, the pixel values ​​of the segmentation maps of each overlapping target subject on any video frame are adjusted.

[0015] Optionally, the method further includes:

[0016] In response to the fact that there are multiple target subjects segmented from the first video frame, the duration corresponding to the longest segmentation graph in the temporal segmentation graph of each target subject is taken as the duration of the video summary.

[0017] Optionally, the method further includes determining the background of the video through the following steps:

[0018] Based on the first video frame and / or each of the second video frames, determine the Gaussian distribution of the pixel values ​​of each pixel in the video;

[0019] Based on the Gaussian distribution corresponding to each pixel in the video, the pixels belonging to the background are identified.

[0020] The background of the video is determined based on the pixels in the background area.

[0021] Optionally, the method further includes:

[0022] According to a preset time period, determine the rate of change between the pixel values ​​of each pixel in the background part of the current period video and the pixel values ​​of each pixel in the background part of the previous period video; the preset time period is shorter than the duration of the video summary.

[0023] In response to a change rate exceeding a preset threshold, the background of the video in the current period is determined based on the pixel values ​​of each pixel in the background portion of the video in the current period.

[0024] If the rate of change does not exceed a preset threshold, the background of the video in the previous period will be used as the background of the video in the current period.

[0025] Secondly, embodiments of this disclosure provide a video summarization generation apparatus, comprising:

[0026] The first segmentation module is used to segment the target subject from the first video frame of the video;

[0027] The second segmentation module is used to segment the target subject into various temporal segmentation maps from each second video frame of the video; the second video frame is the video frame after the first video frame in the video.

[0028] The merging module is used to merge the target subject and its temporal segmentation map with the background of the video in chronological order to obtain a video summary of the video.

[0029] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the first aspect or any optional implementation of the first aspect are performed.

[0030] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any optional implementation thereof.

[0031] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of any of the above embodiments.

[0032] Any of the above-described aspects or any implementation thereof, by segmenting the defective subject from the first video frame and segmenting the target subject temporally from each of the second video frames, obtains temporally segmented images of the target subject using a segmentation method. This improves the accuracy and temporal consistency of the segmented images and better handles occlusion and complex backgrounds. Specifically, the segmentation method operates on a pixel-by-pixel basis, allowing for more precise delineation of the target subject's outline, thereby improving the accuracy of the segmented images and consequently the accuracy of the final generated video summary. Furthermore, the segmentation method captures the precise outline of the target, maintaining consistency even when the target's size, pose, or orientation changes. In contrast, commonly used target detection methods rely on bounding boxes, where deformation or movement can cause jumps or jitters in the detection box position, affecting temporal consistency. By classifying each pixel, the segmentation method can identify unoccluded portions even when the target is partially occluded, avoiding errors caused by scene changes or target occlusion, and improving the applicability and accuracy of the generated video summary. Furthermore, by segmenting the target subject in the first video frame, a temporal segmentation map of the target subject can be obtained in subsequent video frames, which also solves the problem that existing technologies can only track predefined target subjects, further improving the applicability of generating video summaries.

[0033] The effects of the aforementioned video summarization generation device, electronic equipment, and storage medium are described in the description of the aforementioned video summarization generation method, and will not be repeated here. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0035] Figure 1 A flowchart of a video summary generation method provided in an embodiment of this disclosure is shown;

[0036] Figure 2 A schematic diagram illustrating the segmentation of the target body provided in an embodiment of this disclosure is shown;

[0037] Figure 3 A schematic diagram illustrating the segmentation of the target body provided in an embodiment of this disclosure is shown;

[0038] Figure 4 A schematic diagram illustrating the segmentation of the target body provided in an embodiment of this disclosure is shown;

[0039] Figure 5 A schematic diagram illustrating the target subject segmentation result provided in an embodiment of this disclosure is shown;

[0040] Figure 6 A schematic diagram of the temporal segmentation of the target subject provided in an embodiment of this disclosure is shown;

[0041] Figure 7 A schematic diagram illustrating a video summary provided in an embodiment of this disclosure is shown;

[0042] Figure 8 A schematic diagram of a video summarization device provided in an embodiment of this disclosure is shown;

[0043] Figure 9 An exemplary system architecture in which embodiments of this disclosure can be applied is shown;

[0044] Figure 10 A schematic diagram of the structure of a computer system used to implement the embodiments of the present disclosure is shown. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0046] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0047] The above problems and solutions are the result of the inventor's practice and careful research. The discovery process of the above problems and the solutions proposed for the above problems should be considered as the inventor's contribution to this disclosure.

[0048] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely to illustrate selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0049] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0050] To facilitate understanding of this embodiment, a video summarization method disclosed in this disclosure will first be described in detail. The execution entity of the video summarization method provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the video summarization method can be implemented by a processor calling computer-readable instructions stored in memory.

[0051] See Figure 1 The diagram shows a flowchart of a video summary generation method provided in this embodiment of the present disclosure. The method includes steps S101 to S103, wherein:

[0052] S101: Segment the target subject from the first video frame.

[0053] In this embodiment of the disclosure, the target subject can be segmented from the first video frame of the video; the first video frame here can be the first video frame of the video, or any video frame other than the first video frame in the video. Thus, in addition to tracking the target subject appearing in the first video frame, it is also possible to track the target subject appearing in subsequent video frames.

[0054] In one possible implementation, the step of segmenting the target subject from video frames can be performed by a pre-trained large model. Specifically, the video frames used to extract the target subject can be used as input to the large model. The large model can segment all subjects in the video frames by default and then track these subjects in subsequent video frames to generate the final video summary. Alternatively, if the user only focuses on one or a portion of the subjects in the video frame, the large model can segment the target subject in the video frame based on the provided subject descriptive words. Figure 2 As shown, users can input "gray cat" into the large model to segment a gray cat from a video frame as the target subject. Similarly, when the user inputs the subject description "cat" or "white cat and gray cat," the gray cat and white cat can be segmented as the target subjects, ignoring the dog in the frame. To ensure the accuracy of the segmentation results, each pixel in the video frame can be classified to ultimately obtain the target subject.

[0055] Another possible implementation is that the user can segment the target object by directly clicking on it. For example... Figure 3 As shown, users can click Figure 3 The position of the triangular marker divides the vehicle's tires as the main target element. Or, as... Figure 4 As shown, users can also instruct the execution entity of this embodiment to segment the target body by dragging a rectangular frame to select the target body to be segmented. It should be noted that the purpose of dragging the rectangular frame here is to select the target body to be segmented, so there is no specific limitation on the shape of the frame dragged when selecting. In addition to being a rectangle, it can also be any other shape such as a circle.

[0056] It should be noted that the above method for segmenting the target subject from the first video frame is only used as an example of a feasible implementation method of this disclosure and does not constitute an improper limitation of the present invention. In actual applications, the method can be selected or set according to the actual situation. This disclosure does not make specific limitations in this regard, and the method that can achieve its function shall prevail.

[0057] In this embodiment of the disclosure, the process of segmenting the target subject from a video frame can employ segmentation methods such as semantic segmentation or instance segmentation. Semantic segmentation can classify each pixel in the video frame into a specific category, thereby accurately identifying the position and shape of the target subject; while instance segmentation can not only identify specific categories but also distinguish different instances within the same category, ensuring that each target subject has an independent outline and shape. Figure 5 As shown, Figure 5 The image on the left shows the result of semantic segmentation, and the image on the right shows the result of instance segmentation. It should be noted that the above examples are merely illustrative of feasible implementation methods according to the embodiments of this disclosure and do not constitute an improper limitation of the present invention. In practical applications, other arbitrary segmentation methods can also be used to segment the target subject from video frames. The embodiments of this disclosure do not specifically limit this, but rather focus on achieving the desired function.

[0058] S102: Segment the target subject temporally from each of the second video frames of the video; the second video frame is the video frame after the first video frame in the video.

[0059] In this step, after segmenting the target subject from the first video frame, the target subject can be segmented from each subsequent second video frame to obtain various temporal segmentation maps based on the target subject segmented from the first video frame.

[0060] In this embodiment, the number of target subjects can be arbitrary, and the target subjects may not be located in the same frame. For example, assuming there are three subjects in the first video frame, namely subject A, subject B, and subject C, and subject A is taken as the target subject, after segmenting the target subject from the first video frame, subject A can be tracked in subsequent frames of the first video frame to obtain various temporal segmentation maps of subject A. If the subject D of interest appears in any second video frame, then that second video frame can be used as another new first video frame, and subject D can be tracked in subsequent frames after that second video frame. Thus, various temporal segmentation maps of the two target subjects, subject A and subject D, can be obtained from each second video frame.

[0061] In this embodiment of the disclosure, segmenting the target subject from each second video frame in time includes: determining a subject image in the second video frame that includes the target subject; segmenting the subject image corresponding to the second video frame to generate a mask image corresponding to the subject image; wherein the subject image and the mask image have the same size; the pixel value of each pixel in the mask image indicates whether each pixel in the mask image belongs to the target subject; mapping the mask image onto the second video frame according to the position information of the subject image corresponding to the second video frame to determine the segmentation map of the target subject in the second video frame; and sorting the segmentation maps of the target subject in each video frame according to the time sequence of each video frame to obtain the segmentation map of the target subject in time.

[0062] In a specific implementation, it is assumed that the number of target subjects segmented in the first video frame is one. After segmenting the target subject, a rectangle can be used in the first video frame to mark the position and size of the target subject. It should be noted that the rectangle here is an area that includes the target subject. In practical applications, it can also be set to any other shape such as a circle or triangle. This embodiment does not make specific limitations on this, as long as it can achieve its function.

[0063] The execution entity of this embodiment can then extract visual features (such as color features, texture features, etc.) of the target subject from the rectangular frame including the target subject, and extract background features from the background area surrounding the rectangular frame. The target subject and background are distinguished using the data features within the rectangular frame and the features of the background area. By analyzing the aforementioned features contained in the current first video frame, the possible location of the target subject in the next frame can be predicted, and the rectangular frame including the target subject in the next frame can be obtained. After locating the rectangular frame position of the target subject in the next frame, the execution entity of this embodiment updates the corresponding visual features and background features according to the new position, thereby adapting to changes in the target subject that may occur due to lighting, occlusion, etc. Figure 6 As shown, Figure 6The first row of the image includes the identified rectangular bounding box containing the target subject.

[0064] It should be noted that when a new video frame contains a rectangular bounding box of the target subject, it can be determined based on the previous video frame, thereby ensuring the continuity and accuracy of the segmented target subject's various segmentation maps. Alternatively, it can be determined directly based on the first video frame, thereby reducing accumulated errors and adapting to changes in complex scenes. This depends on the specific algorithm and requirements used in the implementation, and this disclosure does not impose specific limitations on this, as long as its functionality can be achieved.

[0065] In one possible implementation, after determining the bounding boxes of the target subject in each second video frame, the regions containing these bounding boxes in each video frame can be cropped to generate a subject image containing the target subject. This allows the computational focus to be concentrated on the region containing the target subject, improving the accuracy and efficiency of segmentation. In practical applications, if the target subject is small, the size of the bounding boxes can be appropriately increased to ensure the integrity of the target subject. Then, the subject images corresponding to each second video frame can be segmented. The execution entity in this embodiment can output a mask image of the same size as the subject image after segmentation. Each pixel value in the mask image can represent whether the pixel belongs to the target subject. After initially obtaining the mask image, further processing can be performed on the mask image to ensure the accuracy of the final segmentation result. Specifically, morphological operations (such as dilation and erosion) can be used to smooth the edges of the segmentation result; alternatively, small, isolated mask regions can be removed from the mask image, retaining the largest connected regions to further optimize the segmentation result. Then, based on the position of the rectangle during cropping, the mask image can be mapped back to the corresponding position in each of the second video frames, thus obtaining a complete segmentation map of the target subject in each of the second video frames. After obtaining the segmentation maps corresponding to each of the second video frames, they can be sorted according to the time sequence of the video to obtain the temporal segmentation maps of the target subject. Figure 6 As shown, Figure 6 The image in the second row represents the temporal segmentation of the target subject.

[0066] Another possible implementation involves using a spatiotemporal convolutional network (combining spatial and temporal convolution to process both spatial and temporal information in video) to perform convolution operations on consecutive frames, simultaneously extracting spatial and temporal features from each second video frame. Through continuous convolution operations, the spatiotemporal convolutional network can capture the temporal motion patterns of the target subject and its spatial distribution in each frame, ultimately outputting segmentation maps of each second video frame. These segmentation maps, arranged chronologically, form the temporal segmentation maps of the target subject. Because the spatiotemporal convolutional network can operate directly in the temporal and spatial domains of the video, it can better capture the dynamic changes of the target subject, enabling the segmentation of the target subject in complex motion scenes.

[0067] Another possible implementation involves using a fully convolutional network (a network structure for image semantic segmentation) and optical flow (used to calculate motion information between adjacent video frames). Combining these two methods can also yield temporal segmentation maps of the target subject. Specifically, a fully convolutional network can first be used to perform semantic segmentation on each second video frame, outputting the segmentation result for each second video frame, i.e., the classification of each pixel in that frame (e.g., the target subject or the background). Then, an optical flow algorithm can be used to calculate pixel motion vectors between adjacent frames, where the pixel motion vector represents the movement path of each pixel from the current frame to the next frame. Using optical flow information, the segmentation result of the previous frame can be mapped to the current frame, thereby correcting the segmentation result of the current frame. Combining the segmentation result obtained through the fully convolutional network and the optical flow information, a temporal segmentation map of the target subject can be obtained. This ensures a smoother segmentation result and maintains the temporal consistency of the target subject.

[0068] It should be noted that the above method for segmenting the target subject in time from each second video frame is only an example of several possible implementation methods of this disclosure. In practical applications, three-dimensional convolutional neural networks, dynamic conditional random fields, and other methods can also be used to obtain the target subject in time in each second video frame. This disclosure does not specifically limit this, but only requires that its function can be achieved.

[0069] It should be noted that if there are multiple target subjects segmented from the first video frame, the above method can be used multiple times to obtain each segmentation map in time corresponding to each target subject; or all target subjects can be obtained in time at once. This embodiment does not specifically limit this.

[0070] S103: Merge the target subject and its temporal segmentation map with the video background in chronological order to obtain a video summary of the video.

[0071] In this step, a video summary can be obtained by combining the target subject and its temporal segmentation map with the video background.

[0072] In this embodiment of the disclosure, the duration of the video summary can be determined first. Specifically, the method further includes: in response to the fact that there are multiple target subjects segmented from the first video frame of the video, the duration corresponding to the longest segmentation map in the temporal segmentation map of each target subject is taken as the duration of the video summary.

[0073] In practice, if there is only one target subject, the duration of the temporal segmentation graph corresponding to that target subject can be directly used as the duration of the video summary. However, when there are multiple target subjects, such as... Figure 7 As shown, the final video summary duration can be determined by the duration of the temporal segmentation graphs of each target subject. For example, if target subject A appears in the video for 60 seconds and target subject B appears for 20 seconds, then the duration of each temporal segmentation graph of target subject A, i.e., 60 seconds, is used as the video summary duration. Alternatively, when the durations of the temporal segmentation graphs of multiple target subjects differ significantly, a compromise can be used to determine the final video summary duration. Furthermore, the video summary duration can be determined based on the movement of the target subject within the frame. For instance, if the temporal segmentation graph of target subject A is 60 seconds long, but target subject A is stationary for 30 seconds, then only the shorter duration of the stationary segmentation graph can be retained. For example, the final video summary duration can be set to 35 seconds, retaining only 5 seconds of the stationary portion, thus both representing the target subject's state and simplifying the video summary.

[0074] It should be noted that the method for determining the duration of the video summary in this embodiment is not specifically limited. In practical applications, it can be set according to the actual situation and needs, so as to achieve its disclosure.

[0075] In one possible implementation, the background of the video summary can be determined before merging the target subject and its temporal segmentation map with the video background. Specifically, the video background is determined through the following steps: based on the first video frame and / or each of the second video frames, a Gaussian distribution of pixel values ​​in the video is determined; based on the Gaussian distribution corresponding to each pixel in the video, pixels belonging to the background portion are determined; and based on the pixels in the background portion, the video background is determined.

[0076] In practical implementation, for each pixel in the video, a Gaussian distribution can be used to model the value changes of each pixel. Each Gaussian distribution can be seen as the range of pixel values ​​under a certain scene, such as the background color under different conditions like daytime, shadow, and nighttime. The Gaussian distribution of each pixel can be represented by the mean, variance, and weights. For example, each Gaussian distribution has a mean to represent the average pixel value of that pixel; a variance to represent the range of pixel value changes; and a weight to represent the proportion of that pixel's Gaussian distribution in the total pixel value changes. For each frame of the video, the existing Gaussian model can be updated based on the new pixel value of each pixel. Specifically, for each pixel, the new pixel value is compared with each of the existing Gaussian distributions to determine whether the difference between the new pixel value and the mean of the corresponding pixel's Gaussian distribution is within the variance range of that distribution. If the new pixel value matches the Gaussian distribution of the corresponding pixel, the Gaussian distribution of that pixel can be updated based on the new pixel value. If they don't match, a new distribution with lower weights can be used to replace the new distribution. This new distribution can use the new pixel value as its mean, a large initial variance, and small weights. This establishes a background model based on the Gaussian distribution. After establishing the background model, pixels corresponding to the Gaussian distribution with higher weights (indicating high and stable frequency) can be used as background pixels, while other pixels can be used as pixels for the main subject. This allows the background of the video to be determined using the background pixels.

[0077] Furthermore, as time goes by and the scene changes, the background will also change, such as lighting, occlusion, and weather changes. At this time, the parameters of the Gaussian distribution can be adjusted to adapt to these changes, thereby accurately separating the background part from the video.

[0078] In another possible implementation, the background of the video can also be determined by methods such as directly setting the background. This disclosure does not specifically limit the method for determining the background of the video, but only requires that the function can be achieved.

[0079] In this embodiment of the disclosure, the video background can be updated at regular intervals. Specifically, the method further includes: determining the rate of change between the pixel values ​​of each pixel in the background portion of the current period's video and the pixel values ​​of each pixel in the background portion of the previous period's video, according to a preset time period; the preset time period is less than the duration of the video summary; in response to the rate of change exceeding a preset threshold, determining the background of the video in the current period based on the pixel values ​​of each pixel in the background portion of the current period's video; and in response to the rate of change not exceeding the preset threshold, using the background of the video in the previous period as the background of the video in the current period.

[0080] In practice, a time period can be set based on the video's duration. Within the first period, the video background can be determined using any of the methods described earlier. Before determining the background for the next period, it can be determined whether to update the background. Specifically, at the beginning of each period, the pixel values ​​of each pixel in the background portion of the video within the current period can be obtained and compared with the pixel values ​​of each pixel in the background portion of the video in the previous period, calculating the rate of change between the two.

[0081] Specifically, a region for pixel value comparison can be predefined. For example, this region can be set as the area around the edges of a video frame. Since the pixels in this area usually do not change significantly with the movement of the target subject, when the rate of change of the pixel values ​​of the pixels in the area around the edges of the frame compared to the previous period exceeds a preset threshold, it can be determined that the background has changed. Furthermore, since a time period can include multiple video frames, the pixel values ​​of each pixel in the background portion of the video in the current period can be obtained by combining the pixel values ​​of all video frames in the period, or the pixel values ​​of the background portion of a single frame can be selected as the pixel values ​​of the video background portion for the entire period; this embodiment does not specifically limit this. The preset threshold can be set according to actual conditions. For example, in practical applications, if it is desired to cover changes in weather, lighting, etc., the preset threshold can be set to a larger value; conversely, if it is desired to promptly detect changes in the surrounding scene, the preset threshold can be set to a smaller value; this embodiment does not specifically limit this.

[0082] Furthermore, the rate of change between the pixel values ​​of each pixel in the current period and those in the previous period can be calculated, and pixels with a rate of change exceeding a preset threshold are marked as change points. When the number of change points exceeds a certain proportion, the background can be considered to have changed. At this point, the background of the video can be determined based on the pixel values ​​of each pixel in the background portion of the video in the current period. Since a time period can include multiple video frames, the pixel values ​​of each pixel in the background portion of the video in a period can be obtained by combining the pixel values ​​of all video frames in the period, or the pixel values ​​of the background portion of a certain frame can be selected as the pixel values ​​of the background portion of the video in the entire period; this disclosure does not specifically limit this, but is based on the ability to achieve its function.

[0083] It is important to note that when setting the time period, the duration of the time period should be shorter than the duration of the final video summary. The time period should be set in a way that avoids wasting too much computing resources due to being too short, while also ensuring that the background is not updated in a timely manner due to being too long.

[0084] When the calculated rate of change exceeds a preset threshold, it indicates that the video background has changed significantly, and the background of the video in the current period can be determined using any of the methods described above. When the calculated rate of change does not exceed the preset threshold, it indicates that the background has not changed significantly, and the background of the previous period can continue to be used as the background of the video in the current period.

[0085] In another possible implementation, the rate of change of pixel values ​​of the background pixels between two adjacent periods may not be calculated; instead, the background of the video may be updated directly when a new period arrives. This disclosure does not specifically limit the method for updating the video background, but rather focuses on achieving the desired functionality.

[0086] In this embodiment of the disclosure, after obtaining the target subject, its temporal segmentation map, and the video background, they can be merged in chronological order to obtain the final video summary. Before merging, if multiple target subjects overlap in the segmentation map of any video frame, the overlapping target subjects can be processed. Specifically, multiple target subjects are segmented from the first video frame; before merging the target subjects and their temporal segmentation maps with the video background in chronological order, the method further includes: adjusting the pixel values ​​of the segmentation maps of the overlapping target subjects in any video frame in response to the overlap of segmentation maps of different target subjects in any video frame.

[0087] In practice, if there are multiple target subjects, overlap may occur in their temporal segmentation maps. To address this, the pixel values ​​of the segmentation maps for each overlapping target subject in the overlapping video frames can be adjusted. For example, the transparency channel of the segmentation map can be increased by a factor of n. To ensure that the overlapping target subjects can be observed, n can be set to a positive number less than 1. Alternatively, the pixel values ​​of the segmentation maps for each overlapping target subject in the overlapping video frames can be directly adjusted to a factor of n. The value of n can be determined based on the number of overlapping target subjects. For example, when two target subjects overlap, n can be set to 0.5; while when four target subjects overlap, n can be set to 0.25.

[0088] It should be noted that the above method for adjusting pixel values ​​is only an example of a possible implementation of the present disclosure and does not constitute an improper limitation of the present invention. In practical applications, other arithmetic methods such as addition or subtraction can also be used to adjust the pixel values ​​of the segmentation map of each target subject on the overlapping video frames. The present disclosure does not specifically limit this, but only requires that its function can be achieved.

[0089] According to a second aspect of the embodiments of this disclosure, such as Figure 8 As shown, a video summarization generation apparatus 800 is provided, comprising:

[0090] The first segmentation module 801 is used to segment the target subject from the first video frame of the video;

[0091] The second segmentation module 802 is used to segment the target subject into various segmentation maps in time sequence from each second video frame of the video; the second video frame is the video frame after the first video frame in the video.

[0092] The merging module 803 is used to merge the target subject and its temporal segmentation map with the background of the video in chronological order to obtain a video summary of the video.

[0093] Optionally, the second segmentation module 802 is specifically used for:

[0094] Identify the subject image in the second video frame that includes the target subject;

[0095] The main image corresponding to the second video frame is segmented to generate a mask image corresponding to the main image; wherein, the main image and the mask image have the same size; the pixel value of each pixel in the mask image indicates whether each pixel in the mask image belongs to the target main body;

[0096] Based on the position information of the subject image corresponding to the second video frame, the mask image is mapped onto the second video frame to determine the segmentation map of the target subject on the second video frame;

[0097] The segmentation maps of the target subject in each video frame are sorted according to the temporal sequence of each video frame to obtain the segmentation maps of the target subject in the temporal sequence.

[0098] Optionally, the target subject segmented from the first video frame can be multiple; the merging module 803 is also used for:

[0099] In response to the overlap of segmentation maps of different target subjects on any video frame, the pixel values ​​of the segmentation maps of each overlapping target subject on any video frame are adjusted.

[0100] Optionally, the merging module 803 is also used for:

[0101] In response to the fact that there are multiple target subjects segmented from the first video frame, the duration corresponding to the longest segmentation graph in the temporal segmentation graph of each target subject is taken as the duration of the video summary.

[0102] Optionally, the device further includes a background determination module 804; the background determination module 804 is specifically used for:

[0103] Based on the first video frame and / or each of the second video frames, determine the Gaussian distribution of the pixel values ​​of each pixel in the video;

[0104] Based on the Gaussian distribution corresponding to each pixel in the video, the pixels belonging to the background are identified.

[0105] The background of the video is determined based on the pixels in the background area.

[0106] Optionally, the background determination module 804 is also used for:

[0107] According to a preset time period, determine the rate of change between the pixel values ​​of each pixel in the background part of the current period video and the pixel values ​​of each pixel in the background part of the previous period video; the preset time period is shorter than the duration of the video summary.

[0108] In response to a change rate exceeding a preset threshold, the background of the video in the current period is determined based on the pixel values ​​of each pixel in the background portion of the video in the current period.

[0109] If the rate of change does not exceed a preset threshold, the background of the video in the previous period will be used as the background of the video in the current period.

[0110] According to a third aspect of the present disclosure, an electronic device for video summarization is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0111] According to a fourth aspect of the present disclosure, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0112] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0113] Figure 9 An exemplary system architecture 900 is shown that can be applied to the video summarization method or video summarization apparatus implemented in this disclosure.

[0114] like Figure 9 As shown, system architecture 900 may include terminal devices 901, 902, and 903, network 904, and server 905. Network 904 is used as a medium to provide a communication link between terminal devices 901, 902, and 903 and server 905. Network 904 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0115] Users can use terminal devices 901, 902, and 903 to interact with server 905 via network 904 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 901, 902, and 903, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0116] Terminal devices 901, 902, and 903 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0117] Server 905 could be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 901, 902, and 903 (for example only). The backend management server can process received video summary generation requests and return the processing results (for example only) to the terminal devices.

[0118] It should be noted that the video summarization method provided in this embodiment of the invention is generally executed by server 905, and correspondingly, the video summarization device is generally located in server 905. The video summarization method provided in this embodiment of the invention can also be executed by terminal devices 901, 902, and 903, and correspondingly, the video summarization device can be located in terminal devices 901, 902, and 903.

[0119] It should be understood that Figure 9 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0120] The following is for reference. Figure 10 It shows a schematic diagram of the structure of a computer system 1000 suitable for implementing a terminal device of the present invention. Figure 10 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0121] like Figure 10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1002 or programs loaded from storage section 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the system 1000. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0122] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0123] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs the functions defined above in the system of this invention.

[0124] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0126] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor. For example, a processor includes a first segmentation module, a second segmentation module, and a merging module. The names of these modules do not necessarily limit the module itself. For example, the first segmentation module can also be described as "segmenting the target subject module from the first video frame of the video".

[0127] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, implement the following method: segmenting a target subject from a first video frame of a video; segmenting the target subject temporally from each of the second video frames of the video; the second video frames being video frames following the first video frame; merging the target subject and its temporally segmented images with the background of the video in chronological order to obtain a video summary of the video.

[0128] Finally, it should be noted that the above embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A video summarization method, characterized in that, include: Segment the target subject from the first video frame; The target subject is segmented temporally from each of the second video frames of the video; the second video frame is the video frame following the first video frame in the video; The target subject and its temporal segmentation map are merged with the background of the video in chronological order to obtain a video summary of the video.

2. The method according to claim 1, characterized in that, Segmenting the target subject into temporal segments from each second video frame of the video, including: Determine that the second video frame includes a subject image of the target subject; The main image corresponding to the second video frame is segmented to generate a mask image corresponding to the main image; wherein the main image and the mask image have the same size; the pixel value of each pixel in the mask image indicates whether each pixel in the mask image belongs to the target subject; Based on the position information of the subject image corresponding to the second video frame, the mask image is mapped onto the second video frame to determine the segmentation map of the target subject on the second video frame; The segmentation maps of the target subject on each video frame are sorted according to the temporal sequence of each video frame to obtain the segmentation maps of the target subject in the temporal sequence.

3. The method according to claim 1, characterized in that, The target subjects segmented from the first video frame are multiple; before merging the target subjects and their temporal segmentation maps with the video background in chronological order, the method further includes: In response to the overlap of segmentation maps of different target subjects on any video frame, the pixel values ​​of the segmentation maps of the overlapping target subjects on that video frame are adjusted.

4. The method according to claim 1, characterized in that, The method further includes: In response to the fact that there are multiple target subjects segmented from the first video frame of the video, the duration corresponding to the longest segmentation map in the temporal segmentation map of each target subject is taken as the duration of the video summary.

5. The method according to claim 1, characterized in that, The method further includes determining the background of the video through the following steps: Based on the first video frame and / or each of the second video frames, determine the Gaussian distribution of the pixel values ​​of each pixel in the video; Based on the Gaussian distribution corresponding to each pixel in the video, the pixels belonging to the background are determined. The background of the video is determined based on the pixels of the background portion.

6. The method according to claim 1, characterized in that, The method further includes: According to a preset time period, determine the rate of change between the pixel values ​​of each pixel in the background portion of the video in the current period and the pixel values ​​of each pixel in the background portion of the video in the previous period; the preset time period is less than the duration of the video summary; In response to the rate of change exceeding a preset threshold, the background of the video in the current period is determined based on the pixel values ​​of each pixel in the background portion of the video in the current period; In response to the change rate not exceeding a preset threshold, the background of the video in the previous period is used as the background of the video in the current period.

7. A video summarization generation device, characterized in that, include: The first segmentation module is used to segment the target subject from the first video frame of the video; The second segmentation module is used to segment the target subject into temporal segmentation maps from each second video frame of the video; the second video frame is a video frame after the first video frame in the video; The merging module is used to merge the target subject and its temporal segmentation map with the background of the video in chronological order to obtain a video summary of the video.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images