A method, device, equipment and storage medium for generating sample data

Abstract

The present disclosure provides a sample data generation method, apparatus, device, and storage medium, relating to the fields of computer technology, particularly to artificial intelligence technologies such as video processing, image processing, and deep learning. A specific implementation scheme comprises: utilizing a shot segmentation model to segment a sample video into multiple shot segments, obtaining multiple shot boundary pseudo-labels, which are used to indicate the shot boundary video frames included in the sample video; deleting a first preset number of consecutive video frames centered around the segmentation point corresponding to the shot boundary pseudo-label; using the last video frame in each shot segment as the true shot boundary label; and using the current sample video and true shot boundary label as sample data for training the target shot segmentation model. Sample data can be automatically generated without the need for manual labeling.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technologies such as video processing, image processing, and deep learning. Background Art

[0002] A complete video is usually composed of multiple shot segments. After identifying the shot segments included in the video, many downstream tasks can be performed based on the recognition results, such as extracting highlight shots, identifying video scenes, and deduplicating video images. Summary of the Invention

[0003] The present disclosure provides a sample data generation method, apparatus, device, and storage medium.

[0004] According to a first aspect of the present disclosure, a method for generating sample data is provided, comprising:

[0005] Using a shot segmentation model to segment a sample video into a plurality of shot segments, a plurality of shot boundary pseudo labels are obtained, wherein the shot boundary pseudo labels are used to indicate shot boundary video frames included in the sample video;

[0006] Deleting a first preset number of consecutive video frames centered around the segmentation point corresponding to the shot boundary pseudo-label;

[0007] The last video frame in each shot segment is used as the ground-truth shot boundary label;

[0008] The current sample video and the real shot boundary label are used as sample data for training the target shot segmentation model.

[0009] According to a second aspect of the present disclosure, there is provided a sample data generating apparatus, comprising:

[0010] a segmentation module, configured to segment the sample video into a plurality of shot segments using a shot segmentation model, and obtain a plurality of shot boundary pseudo labels, wherein the shot boundary pseudo labels are used to indicate shot boundary video frames included in the sample video;

[0011] a deleting module, configured to delete a first preset number of consecutive video frames centered around a segmentation point corresponding to the shot boundary pseudo-label;

[0012] A labeling module that uses the last video frame in each shot segment as the ground-truth shot boundary label;

[0013] The generation module is used to use the current sample video and the real shot boundary label as sample data for training the target shot segmentation model.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is an exemplary schematic diagram of the process of training a deep learning model provided by an embodiment of the present disclosure;

[0023] Figure 2 is a flow chart of a sample data generation method provided by an embodiment of the present disclosure;

[0024] Figure 3a This is an exemplary schematic diagram of shot segments obtained by segmenting using an existing shot segmentation model provided by an embodiment of the present disclosure;

[0025] Figure 3b This is an exemplary schematic diagram of the first effect of processing a shot segment provided by an embodiment of the present disclosure;

[0026] Figure 3c is an exemplary schematic diagram of the second effect of processing a shot segment provided by an embodiment of the present disclosure;

[0027] Figure 3d is an exemplary schematic diagram of the third effect of processing a shot segment provided by an embodiment of the present disclosure;

[0028] Figure 4is an exemplary schematic diagram of the fourth effect of processing a shot segment provided by an embodiment of the present disclosure;

[0029] Figure 5 is a flowchart of another sample data generation method provided by an embodiment of the present disclosure;

[0030] Figure 6 is a structural diagram of a sample data generating device provided by an embodiment of the present disclosure;

[0031] Figure 7 It is a block diagram of an electronic device used to implement the sample data generating method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] The shot segmentation model can be used to identify the shot boundary frames in a video. The shot boundary frame is the last frame in a shot segment, and a video can be segmented into multiple shot segments based on the shot boundary frames. In order for the shot segmentation model to accurately identify the shot boundary frames, it needs to be trained based on a large amount of labeled data.

[0034] like Figure 1 As shown, Figure 1 An exemplary schematic diagram of training a deep learning model as a shot segmentation model is shown. Figure 1 The lower middle section shows a video consisting of four video frames. After this video is input into the deep learning model, the deep learning model can predict whether each video frame is a shot boundary video frame.

[0035] exist Figure 1 In the example, 0 represents a video frame that is not a video boundary frame, and 1 represents a video frame that is a shot boundary frame.

[0036] The deep learning model's prediction result is 0010, meaning the prediction result for the third video frame is 1. This means the deep learning model predicts the third video frame as a shot boundary frame. However, the label (also called annotation) for this video is 0100, meaning the second video frame is actually the shot boundary frame.

[0037] After obtaining the prediction results of the deep learning model, the loss function value can be calculated based on the prediction results and the labels, and the parameters of the deep learning model can be adjusted based on the loss function value. Using a large number of sample videos and labels, a shot segmentation model can be trained.

[0038] It can be seen that in order to obtain a shot segmentation model with high segmentation accuracy, a large amount of sample data is required.

[0039] The present disclosure provides a method for generating sample data, which is applied to electronic devices such as Figure 2 As shown, the method includes:

[0040] S201: Using a shot segmentation model, a sample video is segmented into multiple shot segments to obtain multiple shot boundary pseudo labels, where the shot boundary pseudo labels are used to indicate shot boundary video frames included in the sample video.

[0041] The sample video in the embodiment of the present disclosure may be a TV series, a movie, or other types of short or long videos.

[0042] Among them, the shot segmentation model in this step can be any existing shot segmentation model. The existing shot segmentation model can be a shot segmentation model trained using a small amount of sample data, or a shot segmentation model that can be obtained from a public channel. The segmentation effect accuracy of these shot segmentation models is not high enough. In the embodiment of the present disclosure, the shot boundary video frames identified using these shot segmentation models can be used as shot boundary pseudo labels.

[0043] It can be understood that the shot boundary video frame is the last video frame of each shot segment in the multiple shot segments.

[0044] As an example, Figure 3a As shown, Figure 3a The last video frame in each row is the shot boundary video frame predicted by the shot segmentation model. Figure 3a It can be seen that, in fact, the first video frame in the second row and the first video frame in the third row are the actual shot boundary video frames.

[0045] S202: Delete a first preset number of consecutive video frames centered around the segmentation points corresponding to the shot boundary pseudo labels.

[0046] In one embodiment, at least one shot boundary pseudo label may be randomly selected from the multiple shot boundary pseudo labels obtained in S201, and for each selected pseudo label, a first preset number of consecutive video frames centered around the segmentation point corresponding to the shot boundary pseudo label are deleted.

[0047] In another embodiment, for each shot boundary pseudo label among the multiple shot boundary pseudo labels obtained in S201, a first preset number of consecutive video frames centered around the segmentation point corresponding to the shot boundary pseudo label may be deleted.

[0048] The segmentation point corresponding to the shot boundary pseudo label refers to the position between the shot boundary pseudo label and the next video frame in the sample video, for example Figure 3a Select the rightmost black line in the image and split the video at the split point to divide the sample video into multiple shot segments.

[0049] The first preset number may be a preset fixed number, or may be a specified ratio of the number of video frames included in two adjacent shot segments.

[0050] For example, if the number of video frames included in two adjacent shot segments is 8 and the specified ratio is 25%, then the first preset number is 2.

[0051] like Figure 3b As shown, in Figure 3a On the basis of Figure 3b Delete one video frame before and one video frame after each black line on the far right. Figure 3b The video frames that are blocked in the image are the deleted video frames.

[0052] Will Figure 3a and Figure 3b By comparison, Figure 3a The last video frame in the first row and the first video frame in the second row actually belong to the same shot, that is, the shot boundary pseudo-labels identified by the shot segmentation model are not accurate enough. Figure 3b In the figure, the video frames in the rectangular brackets are the remaining video frames. It can be seen that the differences between the remaining video frames in the first row, the remaining video frames in the second row, and the remaining video frames in the third row are relatively large, and they do belong to different shot segments. Therefore, the third video frame in the first row and the third video frame in the second row can both be used as real shot boundary labels.

[0053] S203: Use the last video frame in each shot segment as a true shot boundary label.

[0054] The true shot boundary label is used to indicate the actual shot boundary video frame of each shot segment included in the sample video, that is, to indicate an accurate labeling result.

[0055] It can be understood that the last video frame in each shot segment refers to the last video frame remaining in the shot segment after executing S202.

[0056] S204: Use the current sample video and the real shot boundary label as sample data for training the target shot segmentation model.

[0057] The current sample video refers to the video obtained after processing the initial sample video. For example, Figure 3b The video composed of the video frames that have not been deleted is the current sample video.

[0058] By adopting the above technical solution, a shot segmentation model is used to obtain multiple shot boundary pseudo-labels of a sample video. The shot boundary video frames identified by the shot segmentation model have certain errors. Therefore, a first preset number of continuous video frames centered on the segmentation point corresponding to the shot boundary pseudo-label can be deleted. Since the position where the shot segmentation model inaccurately identifies is usually the video frame at the shot transition, deleting the video frames near the segmentation point is equivalent to deleting the video frames at the shot transition. In this way, the remaining video frames in adjacent shot segments do not belong to the same shot. Then, the last video frame in each shot segment can be used as the true shot boundary label, and the current sample video and the true shot boundary label can be used as sample data for training the target shot segmentation model. The above process does not require human intervention, can realize automatic labeling and automatic generation of sample data, and thus does not require manual labeling of a large number of videos, can reduce the cost of obtaining sample data, and improve the efficiency of sample data generation.

[0059] In some embodiments, before using the current sample video and the real shot boundary label as sample data for training the target shot segmentation model, the method further includes:

[0060] For any two adjacent shot clips among the multiple shot clips, one of the shot clips is randomly deleted.

[0061] Optionally, the electronic device may randomly delete one of the two adjacent shot segments after S203, or may randomly delete one of the two adjacent shot segments before S202.

[0062] Taking the deletion before S202 as an example, then Figure 3a On the basis of Figure 3c As shown, the video frames included in a middle shot segment can be deleted. Figure 3c The video frames covered in the figure are deleted video frames, and the video frames in the rectangular brackets are the remaining video frames.

[0063] Using this method, since the video frames in adjacent shot segments in the original sample video may be relatively similar, one of the adjacent shot segments can be randomly deleted. In this way, the similarity between the remaining adjacent shot segments is reduced, which can further improve the accuracy of the obtained real shot boundary labels.

[0064] In some embodiments of the present disclosure, before using the current sample video and the real shot boundary labels as sample data for training the target shot segmentation model, the plurality of shot segments may be reordered.

[0065] The reordering process may be performed before S202 , or after S203 , or after randomly deleting one of any two adjacent shot segments among the multiple shot segments.

[0066] There are three ways to reorder multiple shot clips:

[0067] Method 1: Randomly reorder multiple shot clips.

[0068] For example, taking the reordering process as an example, that is, Figure 3a The three shot clips in are randomly reordered, and the results of random reordering are as follows Figure 3d As shown, it can be seen Figure 3a The order of the first shot clip and the second shot clip in are swapped.

[0069] Since the images between adjacent shot segments are relatively similar, inaccurate segmentation is prone to occur. Randomly reordering multiple shot segments is relatively simple to implement, and the image differences between adjacent shot segments after reordering can be increased, which can make the subsequent real shot boundary labels more accurate.

[0070] Method 2: Divide the sample video into multiple video segments evenly, and randomly reorder the shot segments in at least one video segment.

[0071] The electronic device may divide the sample video equally according to the number of shot clips. For example, if the sample video includes 100 shot clips, the sample video may be divided equally into 10 video segments, each of which includes 10 shot clips.

[0072] After obtaining the video segments, at least one video segment may be randomly selected, and the shot segments in each selected video segment may be randomly reordered; or the shot segments in each obtained video segment may be randomly reordered.

[0073] For example, 3 video segments may be randomly selected from the 10 video segments, and the 10 shot segments in each of the 3 selected video segments may be randomly reordered;

[0074] Alternatively, the 10 shot segments in each of the 10 video segments are randomly reordered.

[0075] For longer videos, there may be multiple shot transitions within a continuous period of time. Therefore, the sample video can be segmented first, and then the shot segments can be randomly reordered within at least one video segment. This can also achieve the effect of dispersing shot segments with similar images, making the subsequent real shot boundary labels more accurate.

[0076] Method 3: reorder multiple shot segments based on the similarity between the multiple shot segments.

[0077] By reordering multiple shot segments based on the similarity between the shot segments, we can more accurately determine whether the images between the shot segments are similar, thereby more precisely adjusting the order between the shot segments, making the subsequent real shot boundary labels more accurate.

[0078] Method 3 can be specifically implemented by randomly selecting a shot segment to be sorted from multiple shot segments; selecting a shot segment from the shot segments after the shot segment to be sorted, and moving the selected shot segment to a position after the shot segment to be sorted and adjacent to the shot segment to be sorted. The similarity between the shot segment to be sorted and the selected shot segment is within a preset range.

[0079] Optionally, one or more shot segments may be randomly selected from the plurality of shot segments as the shot segments to be sorted, or each shot segment may be selected as the shot segment to be sorted.

[0080] It can be understood that a video is composed of video frames, each shot segment includes multiple video frames, and each video frame in the video corresponds to a timestamp. In the embodiment of the present disclosure, the timestamps of the video frames can be used to distinguish the sequence between shot segments, and the video frames with smaller timestamps are arranged closer to the front.

[0081] Alternatively, a video may be viewed as a video frame sequence composed of video frames, and the arrangement order of the video frames in the video frame sequence may be used to represent the arrangement order between shot segments.

[0082] For example, the time interval corresponding to the video frames included in shot segment 1 is 5:00-5:20, the time interval corresponding to the video frames included in shot segment 2 is 5:21-5:55, and the time interval corresponding to the video frames included in shot segment 3 is 4:20-4:50, then shot segment 3 is before shot segment 1, and shot segment 1 is before shot segment 2.

[0083] For example, for a shot segment to be sorted, the similarity between the shot segment to be sorted and the first shot segment after it is calculated. If the similarity is within a preset range, the order of the two adjacent shot segments does not need to be adjusted. If the similarity is not within the preset range, the similarity between the shot segment to be sorted and the second shot segment after it is calculated.

[0084] If the recalculated similarity is within the preset range, the second shot segment after the shot segment to be sorted is moved to the position adjacent to the shot segment to be sorted. If the similarity is still not within the preset range, the similarity between the shot segment to be sorted and the third shot segment after it is calculated again until a shot segment with a similarity within the preset range is found.

[0085] Then, the same process may be performed on the next shot segment to be sorted.

[0086] The preset range may be a range represented by (low_threhold, high_threshold), wherein low_threhold is the lowest similarity threshold and high_threshold is the highest similarity threshold.

[0087] In the disclosed embodiment, the method for calculating the similarity between shot segments is as follows: respectively extracting key frames from two shot segments, then calculating the image similarity between the key frames extracted from the two shot segments, and using the calculated image similarity as the similarity between the two shot segments.

[0088] For each shot segment, you can evenly extract video frames from the segment and use them as key frames for the segment. For example, for a 10-second segment, you can extract the video frames at the 1st second, the 5th second, and the 9th second, and use them as key frames for the segment.

[0089] Alternatively, five video frames may be extracted, for example, video frames at the 1st, 3rd, 5th, 7th, and 9th seconds may be extracted as key frames.

[0090] Then, the similarity between the key frames extracted from the two shot segments can be calculated using deep learning or the color histogram features of the image. The method for calculating image similarity can adopt any method in the relevant technology, which is not limited in the embodiments of the present disclosure.

[0091] Because a shot segment is composed of multiple video frames, the image similarity between key frames in two shot segments calculated in the disclosed embodiments accurately reflects whether the images in the two shot segments are similar. This image similarity is then used as the similarity between the shot segments. Subsequently, sample videos can be adjusted based on this accurate similarity between the shot segments, resulting in high-quality sample data.

[0092] By placing a shot segment with a similarity within a preset range after each shot segment, the similarity between adjacent shot segments in the resulting sample video can be kept neither too high nor too low. This prevents the target shot segmentation model from being too simple or too complex, resulting in higher-quality sample data. This allows for the generation of high-quality sample data without the need for manual labeling.

[0093] In another embodiment, the electronic device may first evenly divide the sample video into multiple video segments, and then perform the above-mentioned method 3 processing on the shot segments in at least one video segment.

[0094] Based on any of the above embodiments, before S204, using the current sample video and the real shot boundary labels as sample data for training the target shot segmentation model, the method further includes:

[0095] Shot segments of a first preset ratio are randomly selected from the plurality of shot segments, and for each selected shot segment, a gradual change process is performed on the video frames included in the shot segment.

[0096] For example, if the total number of shot segments is 100, the first preset ratio is 0.5, and the calculated second number is 50, 50 shot segments are randomly selected for gradient processing.

[0097] The gradient processing may be brightness gradient, blur gradient, and gradual segmentation up, down, left, and right, etc. The specific method of the gradient processing is not limited in the embodiment of the present disclosure.

[0098] As an example, Figure 4 As shown, Figure 4 The first row of video frames in the figure shows the effect after blurring, that is, gradually becoming clearer from left to right. Since the video frames in the first row are obtained by transforming the same shot, the label values of the first row of video frames are (0, 0, 0, 0, 0).

[0099] By adopting the above method, the video frames in some shot segments in the sample video can be gradient processed, so that the generated sample data can adapt to the situation where the shot in the actual video has gradient, and high-quality sample data can be automatically obtained, so that the target video segmentation model trained with the sample data can accurately segment the shot segments with gradient effects.

[0100] Based on any of the above embodiments, before S204, using the current sample video and the real shot boundary labels as sample data for training the target shot segmentation model, the method further includes:

[0101] A second preset ratio group of adjacent shot segments is randomly selected from the plurality of shot segments, and for each selected group of adjacent shot segments, a weighted fusion process is performed on a second preset number of video frames in the group of adjacent shot segments, centered around the last video frame of the preceding shot segment.

[0102] For example, the first and second shot clips in the sample video are taken as a group, the third and fourth shot clips are taken as a group, and so on. Assuming that the sample video includes 50 shot clip groups, if the second preset ratio is 10%, 5 groups of adjacent shot clips are randomly selected from the sample video, and the video frames in each group of adjacent shot clips are weighted fused.

[0103] The second preset number can be determined based on the number of video frames included in a group of adjacent shot segments. For example, if shot segment 1 includes 5 video frames and shot segment 2 includes 5 video frames, then the number of video frames included in the group of adjacent shot segments is 10. 40% of the video frames can be selected for weighted fusion processing, that is, the second preset number is 4. Then, the last two video frames in shot segment 1 (assuming video frames 4 and 5) can be weighted fused with the last two video frames in shot segment 2 (assuming video frames 6 and 7).

[0104] The weight of each video frame during weighted fusion is related to the position of the video frame in the shot segment. For video frames in two adjacent shot segments, when weighted fusion is performed, the weight of the front video frame decreases and the weight of the back video frame increases in order from front to back.

[0105] For example, in the above example, we can perform a weighted fusion on video frames 4 and 7, where the weight of video frame 4 is higher than the weight of video frame 7, and then replace the original video frame 4 with the video frame obtained after weighted fusion. Similarly, we can perform a weighted fusion on video frames 4 and 7, where the weight of video frame 7 is higher than the weight of video frame 4, and then replace the original video frame 7 with the video frame obtained after weighted fusion.

[0106] As an example, Figure 4 As shown, in Figure 4 In the second row of video frames, the first two video frames in the second row belong to shot 1, and the last three video frames belong to shot 2. That is, the label values of the second row of video frames are (0, 1, 0, 0, 0). Figure 4 The video frames in the square brackets in the second row are the video frames after weighted fusion.

[0107] It should be noted that, in the embodiment of the present disclosure, the weighted fusion process may be performed after the real shot boundary labels are determined.

[0108] By adopting the above method, by performing weighted fusion processing on the video frames in the adjacent lens group shot segments, the video frames after weighted fusion processing can be made to present the effect of gradually transitioning from one picture to another, so that the generated sample data can adapt to the situation where the shots in the actual video have a gradual transition, which can increase the complexity of the target video segmentation model in identifying the shot boundary and improve the recognition effect of the trained target segmentation model.

[0109] like Figure 5 As shown, Figure 5 A flow chart of a method for generating sample data provided in an embodiment of the present disclosure, the method comprising:

[0110] S501: Obtain an unlabeled video.

[0111] S502: Obtain shot boundary pseudo labels using a shot segmentation model.

[0112] The existing shot segmentation model is used to segment the unlabeled video, and the last video frame of each shot obtained by segmentation is used as the shot boundary pseudo label.

[0113] S503: Remove shot boundaries, remove intermediate shots, and rearrange shots.

[0114] The method for removing the shot boundary may refer to the relevant description in S202 above.

[0115] Removing the middle shot means that, for any two adjacent shot segments in the multiple shot segments, one of the shot segments is randomly deleted.

[0116] The method of rearranging shots may refer to the method of reordering shot segments described in the above embodiment.

[0117] S504: Lens change.

[0118] The shot transformation refers to performing a gradual transformation process on the video frames included in the same shot segment, and performing a weighted fusion process on the video frames included in adjacent shot segments. Please refer to the relevant description in the above embodiment.

[0119] S505: Generate sample data.

[0120] After the above series of processing, the final video can be used as a sample video, and the real boundary label can be used as the annotation data of the sample video, so as to obtain sample data for training the target shot segmentation model, realizing the automatic generation of high-quality sample data and greatly reducing labor costs.

[0121] Based on the same concept, the embodiment of the present disclosure provides a sample data generating device, such as Figure 6 As shown, including:

[0122] The segmentation module 601 is configured to segment the sample video into multiple shot segments using a shot segmentation model, and obtain multiple shot boundary pseudo labels, where the shot boundary pseudo labels are used to indicate shot boundary video frames included in the sample video.

[0123] The deletion module 602 is configured to delete a first preset number of consecutive video frames centered around the segmentation point corresponding to the shot boundary pseudo-label.

[0124] The labeling module 603 is configured to use the last video frame in each shot segment as a real shot boundary label.

[0125] The generation module 604 is configured to use the current sample video and the real shot boundary label as sample data for training the target shot segmentation model.

[0126] Optionally, the deleting module 602 is further configured to randomly delete one of any two adjacent shot segments among the multiple shot segments.

[0127] Optionally, the apparatus further comprises a reordering module;

[0128] A reordering module, used to randomly reorder multiple shot clips; or,

[0129] Divide the sample video into multiple video segments evenly, and randomly reorder the shot segments in at least one video segment; or

[0130] The multiple shot segments are reordered based on similarities between the multiple shot segments.

[0131] Optionally, the reordering module is specifically configured to:

[0132] A shot segment to be sorted is randomly selected from a plurality of shot segments.

[0133] A shot segment is selected from the shot segments after the shot segment to be sorted, and the selected shot segment is moved to a position after the shot segment to be sorted and adjacent to the shot segment to be sorted; wherein the similarity between the shot segment to be sorted and the selected shot segment is within a preset range.

[0134] Optionally, the reordering module is further configured to obtain similarities between shot segments in the following manner:

[0135] Extract key frames from the two shot clips respectively.

[0136] The image similarity between the key frames extracted from the two shot segments is calculated, and the calculated image similarity is used as the similarity between the two shot segments.

[0137] Optionally, the device further comprises:

[0138] The gradient module is used to randomly select a shot segment of a first preset ratio from the plurality of shot segments, and perform gradient processing on the video frames included in each selected shot segment.

[0139] Optionally, the device further comprises:

[0140] The weighted fusion module is used to randomly select a second preset ratio of adjacent shot segments from multiple shot segments, and for each selected group of adjacent shot segments, perform weighted fusion processing on a second preset number of video frames in the group of adjacent shot segments centered on the last video frame of the preceding shot segment.

[0141] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0142] It should be noted that the sample videos in this embodiment come from a public dataset.

[0143] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0144] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0145] like Figure 7 As shown, electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of electronic device 700 may also be stored in RAM 703. Computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0146] Multiple components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0147] The computing unit 701 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the sample data generation method. For example, in some embodiments, the sample data generation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the sample data generation method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the sample data generation method via any other suitable means (e.g., via firmware).

[0148] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0153] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0154] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0155] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating sample data, comprising: Using a shot segmentation model, the sample video is segmented into a plurality of shot segments to obtain a plurality of shot boundary pseudo labels, wherein the shot boundary pseudo labels are used to indicate a shot boundary video frame included in the sample video, and the shot boundary video frame is the last video frame in a shot segment; Deleting a first preset number of consecutive video frames centered around a segmentation point corresponding to the shot boundary pseudo-label, where the segmentation point corresponding to the shot boundary pseudo-label is a position between the shot boundary pseudo-label and the next video frame in the sample video; The last video frame in each shot segment is used as the true shot boundary label, and the last video frame in each shot segment is the last video frame remaining in the shot segment; The current sample video and the real shot boundary label are used as sample data for training the target shot segmentation model.

2. The method according to claim 1, before using the current sample video and the real shot boundary labels as sample data for training the target shot segmentation model, the method further comprises: For any two adjacent shot segments among the multiple shot segments, one of the shot segments is randomly deleted.

3. The method according to claim 1 or 2, before using the current sample video and the real shot boundary labels as sample data for training the target shot segmentation model, the method further comprises: Randomly reordering the multiple shot clips; or, Divide the sample video into multiple video segments on average, and randomly reorder the shot segments in at least one video segment; or The plurality of shot segments are reordered based on similarities between the plurality of shot segments.

4. The method according to claim 3, wherein: The reordering of the plurality of shot segments based on the similarities between the plurality of shot segments includes: Randomly selecting a shot segment to be sorted from the multiple shot segments; A shot segment is selected from the shot segments after the shot segment to be sorted, and the selected shot segment is moved to a position after the shot segment to be sorted and adjacent to the shot segment to be sorted; wherein the similarity between the shot segment to be sorted and the selected shot segment is within a preset range.

5. The method according to claim 4, wherein The similarity between shot segments is obtained in the following way: Extract key frames from two shot clips respectively; The image similarity between the key frames extracted from the two shot segments is calculated, and the calculated image similarity is used as the similarity between the two shot segments.

6. The method according to claim 1, before using the current sample video and the real shot boundary labels as sample data for training the target shot segmentation model, the method further comprises: Shot segments of a first preset ratio are randomly selected from the multiple shot segments, and for each selected shot segment, a gradual change process is performed on the video frames included in the shot segment.

7. The method according to claim 1 or 6, before using the current sample video and the real shot boundary labels as sample data for training the target shot segmentation model, the method further comprises: A second preset ratio group of adjacent shot segments is randomly selected from the multiple shot segments, and for each selected group of adjacent shot segments, a weighted fusion process is performed on a second preset number of video frames in the group of adjacent shot segments, centered around the last video frame of the preceding shot segment.

8. A sample data generating device, comprising: a segmentation module, configured to segment the sample video into a plurality of shot segments using a shot segmentation model, and obtain a plurality of shot boundary pseudo labels, wherein the shot boundary pseudo labels are used to indicate shot boundary video frames included in the sample video, and the shot boundary video frame is the last video frame in a shot segment; a deleting module, configured to delete a first preset number of consecutive video frames centered around a segmentation point corresponding to the shot boundary pseudo-label, wherein the segmentation point corresponding to the shot boundary pseudo-label is a position between the shot boundary pseudo-label and the next video frame in the sample video; A labeling module is used to use the last video frame in each shot segment as a true shot boundary label, and the last video frame in each shot segment is the last video frame remaining in the shot segment; The generation module is used to use the current sample video and the real shot boundary label as sample data for training the target shot segmentation model.

9. The device according to claim 8, The deleting module is further configured to randomly delete one of any two adjacent shot segments among the multiple shot segments.

10. The apparatus according to claim 8 or 9, further comprising a reordering module; The reordering module is configured to randomly reorder the plurality of shot segments; or Divide the sample video into multiple video segments on average, and randomly reorder the shot segments in at least one video segment; or The plurality of shot segments are reordered based on similarities between the plurality of shot segments.

11. The device according to claim 10, wherein The reordering module is specifically used to: Randomly selecting a shot segment to be sorted from the multiple shot segments; A shot segment is selected from the shot segments after the shot segment to be sorted, and the selected shot segment is moved to a position after the shot segment to be sorted and adjacent to the shot segment to be sorted; wherein the similarity between the shot segment to be sorted and the selected shot segment is within a preset range.

12. The device according to claim 11, wherein The reordering module is further configured to obtain the similarity between shot segments in the following manner: Extract key frames from two shot clips respectively; The image similarity between the key frames extracted from the two shot segments is calculated, and the calculated image similarity is used as the similarity between the two shot segments.

13. The apparatus according to claim 8, further comprising: The gradient module is configured to randomly select a shot segment of a first preset ratio from the plurality of shot segments, and perform gradient processing on the video frames included in each selected shot segment.

14. The device according to claim 8 or 13, further comprising: The weighted fusion module is configured to randomly select a second preset ratio of groups of adjacent shot segments from the plurality of shot segments, and for each selected group of adjacent shot segments, perform weighted fusion processing on a second preset number of video frames in the group of adjacent shot segments, centered around the last video frame of the preceding shot segment.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images