Video segment sequence generation method and device and electronic equipment

Abstract

The invention provides a video segment sequence generation method and device and electronic equipment, and the method comprises the steps: responding to a video segment sequence generation request, analyzing a target video, and extracting a picture stream and an audio stream; performing frame extraction on the picture stream according to a preset first duration to generate a picture frame and a timestamp thereof; performing alignment frame extraction on the audio stream based on the timestamp and a preset second duration to generate a target audio frame; and fusing the target audio frame and the picture frame to generate a video frame, and clustering the video frame to generate a video segment sequence.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular to a method, apparatus and electronic device for generating video segment sequences. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this application as set forth in the claims. The description herein is not to be construed as prior art simply because it is included in this section.

[0003] With the continued explosive growth of user-generated content (UGC), video platforms face the challenge of managing massive, diverse, and dynamically changing content. Developing video scene recognition technology is crucial for the intelligent management of the content ecosystem. By identifying scenes in videos in real time, such as games, animation, or education, platforms can more accurately categorize content and optimize recommendation algorithms, thereby improving the efficiency with which users discover videos of interest.

[0004] However, current video scene recognition technologies primarily rely on two methods: unimodal feature recognition and multimodal fusion, both of which have significant drawbacks. Traditionally, unimodal feature recognition, based on visual or audio features from video frames, such as support vector machines or random forests, can accomplish scene classification tasks to some extent, but is prone to misclassification in complex and dynamic scenes. Deep learning techniques, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have further improved the extraction capabilities of unimodal features, but still face challenges in handling complex and dynamic scenes. While multimodal fusion methods attempt to combine visual and audio features, they typically employ only simple feature concatenation or weighted fusion strategies, failing to fully exploit the semantic relationships between modalities, especially regarding insufficient modeling of temporal synchronization and contextual relationships, resulting in limited recognition accuracy in complex scenes.

[0005] Based on this, this application urgently needs to propose a video segment sequence generation method and device to solve the technical problems in the prior art where single-modal information is difficult to fully represent scene semantics, and traditional multimodal fusion methods lack effective modeling of key factors, thus limiting the recognition accuracy in complex scenes. Summary of the Invention

[0006] This application provides a video segment sequence generation method, device, and computer-readable storage medium to address the technical problems in the prior art where single-modal information is difficult to fully represent scene semantics, and traditional multimodal fusion methods lack effective modeling of key factors, thus limiting the recognition accuracy in complex scenes.

[0007] Firstly, this application proposes a method for generating video segment sequences, the method comprising: In response to a video segment sequence generation request, the target video is parsed to extract the video and audio streams; The video stream is frame-by-frame extracted according to a preset first duration to generate video frames and their timestamps. Based on the timestamp and the preset second duration, the audio stream is aligned and frames are extracted to generate the target audio frame; The target audio frame and the picture frame are merged to generate a video frame, and the video frames are clustered to generate a video segment sequence.

[0008] Further, the process of clustering the video frames to generate a video segment sequence includes: obtaining the first na video frames and the first n+a video frames based on the nth video frame; obtaining the type probability value of each scene type of the target video segment based on a preset scene type; determining the maximum value of the type probability value and its corresponding scene type as the type probability value and scene type of the nth video frame; sequentially clustering video frames of the same scene type into video segments, and generating a video segment sequence based on the video segments; wherein, the target video segment includes the nth video frame, the first na video frames, and the first n+a video frames, where n > a, and n and a are both positive integers.

[0009] Furthermore, the sequential clustering of video frames of the same scene type into video segments includes: obtaining the scene type of the nth video frame and the scene type of the (n-1)th video frame respectively; when the scene type of the nth video frame is consistent with the scene type of the (n-1)th video frame, clustering the video segments to which the nth video frame to the (n-1)th video frame belong.

[0010] Furthermore, the method further includes: when the scene type of the nth video frame is inconsistent with the scene type of the (n-1)th video frame, generating a new video segment based on the nth video frame; and determining the scene type of the nth video frame as the scene type of the video segment.

[0011] Furthermore, the method further includes: when there are multiple video segments, determining the order of the video segments based on the timestamps of the video segments; when the duration of the m-th video segment is not greater than a preset duration, obtaining a first scene type and a second scene type respectively; determining the scene type of the m-th video segment based on the first scene type and the second scene type; wherein, the first scene type is the scene type of the (m-1)-th video segment, and the second scene type is the scene type of the (m+1)-th video segment.

[0012] Further, determining the scene type of the m-th video segment based on the first scene type and the second scene type includes: when the first scene type and the second scene type are inconsistent, determining the type probability value of the m-th video segment corresponding to the first scene type as a first probability value, determining the type probability value of the m-th video segment corresponding to the second scene type as a second probability value, determining the type probability value of the first scene type as a third probability value, and determining the type probability value of the second scene type as a fourth probability value; when the difference between the first probability value and the third probability value is greater than the difference between the second probability value and the fourth probability value, determining the second scene type as the scene type of the m-th video segment; when the difference between the first probability value and the third probability value is less than the difference between the second probability value and the fourth probability value, determining the first scene type as the scene type of the m-th video segment.

[0013] Furthermore, the step of obtaining the type probability value of each scene type of the target video segment based on the preset scene type includes: obtaining the type probability value of each video frame of the target video segment based on each scene type; Based on each scene type and the type probability value of each video frame, obtain the type probability value of each scene type in the target video segment.

[0014] Further, the audio stream is aligned and frames are extracted based on the timestamp and the preset second duration to generate a target audio frame, including: determining an audio frame with a second duration x times before the timestamp as the first audio frame and an audio frame with a second duration y times after the timestamp as the second audio frame; generating the target audio frame based on the first audio frame and the second audio frame.

[0015] Furthermore, generating a video segment sequence based on the video segments includes: obtaining the scene type of each video segment; clustering video segments with the same scene type to generate a video segment sequence; wherein, when there are multiple video segments, the timestamp of the previous video segment in the video segment sequence is earlier than the timestamp of the next video segment.

[0016] Secondly, this application provides a video segment sequence generation apparatus, the apparatus comprising: The parsing unit is used to parse the target video to extract the video stream and audio stream in response to a video segment sequence generation request; The frame extraction unit is used to extract frames from the video stream according to a preset first duration to generate video frames and their timestamps, and to perform aligned frame extraction from the audio stream based on the timestamps and a preset second duration to generate target audio frames. A fusion unit is used to fuse the target audio frame and the picture frame to generate a video frame, and to cluster the video frames to generate a video segment sequence.

[0017] Thirdly, an electronic device is provided, the device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the video segment sequence generation method as described above.

[0018] Fourthly, a computer-readable storage medium is provided, on which computer program instructions are stored, which can be executed by a processor to implement the video segment sequence generation method described above.

[0019] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements the video segment sequence generation method.

[0020] This application proposes a video segment sequence generation method, apparatus, and electronic device. The method includes, in response to a video segment sequence generation request, parsing a target video to extract a video stream and an audio stream; extracting frames from the video stream according to a preset first duration to generate video frames and their timestamps; aligning and extracting frames from the audio stream based on the timestamps and a preset second duration to generate target audio frames; fusing the target audio frames and the video frames to generate video frames; and clustering the video frames to generate a video segment sequence. This method improves the accuracy of scene recognition by performing scene recognition after aligning and extracting the video stream and audio stream of the target video; simultaneously, aligning and extracting frames from the audio stream based on the timestamps and the preset second duration to generate target audio frames corresponding to the video frames, realizing dynamic multimodal context-aligned video frame segmentation, and overcoming the technical defects of audio and video semantic separation in the prior art; it overcomes the technical disadvantages of single-modal information being unable to fully represent scene semantics and multimodal fusion methods being unable to accurately identify complex scenes due to a lack of time synchronization and contextual relevance. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1A flowchart illustrating a video segment sequence generation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a target video segment in a video segment sequence generation method provided in an embodiment of this application; Figure 3 A schematic diagram of a video segment sequence generation apparatus provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device suitable for implementing the solutions in the embodiments of this application. The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0025] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0026] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0027] It should be noted that, in any aspect of this application involving data collection, storage, use, transmission, and processing, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0028] This application provides a method for generating video segment sequences, the method comprising: Step S101: In response to the video segment sequence generation request, parse the target video to extract the video stream and audio stream; Step S102: Extract frames from the video stream according to the preset first duration to generate video frames and their timestamps; Step S103: Align and extract frames from the audio stream based on the timestamp and the preset second duration to generate the target audio frame; Step S104: Fuse the target audio frame and the picture frame to generate a video frame, and cluster the video frames to generate a video segment sequence.

[0029] In practical scenarios, the execution entity of this method can be a user device, or a device formed by integrating a user device and a network device through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0030] Figure 1The present application illustrates a processing flow of a video segment sequence generation method according to an embodiment of this application. The method includes at least the following processing steps: Step S101: In response to the video segment sequence generation request, parse the target video to extract the video stream and audio stream; Step S102: Extract frames from the video stream according to the preset first duration to generate video frames and their timestamps; Step S103: Align and extract frames from the audio stream based on the timestamp and the preset second duration to generate the target audio frame; Step S104: Fuse the target audio frame and the picture frame to generate a video frame, and cluster the video frames to generate a video segment sequence.

[0031] In one embodiment, the first duration is 0.5s-1s; preferably, the first duration is 0.5s.

[0032] In one embodiment, the second duration is 1s-3s; preferably, the second duration is 2s.

[0033] It should be understood that this application does not limit the specific values ​​of the first duration and the second duration, and those skilled in the art can make reasonable choices according to the actual application scenario.

[0034] In one embodiment, the video stream is frame-by-frame extracted according to a preset first duration to generate a video frame and its timestamp, including: extracting a video frame every 0.5 seconds to generate a video frame, and determining the time point of the video frame in the video stream as its timestamp; for example, if the video frame is at 2 minutes and 15 seconds in the video stream, then the timestamp of the video frame is 2 minutes and 15 seconds.

[0035] Furthermore, based on the timestamp and a preset second duration, the audio stream is aligned and frames are extracted to generate a target video frame. This includes: determining, based on the timestamp, an audio frame with a second duration x times the timestamp as the first audio frame and an audio frame with a second duration y times the timestamp as the second audio frame; generating the target audio frame based on the first and second audio frames. For example, when x and y are equal and both x and y are 1, and the second duration is 2 seconds, then starting from 2 minutes and 15 seconds, the audio frames corresponding to 2 minutes and 13 seconds - 2 minutes and 15 seconds and 2 minutes and 15 seconds - 2 minutes and 17 seconds are obtained as the first audio frame and the second video frame, respectively. Based on the dynamic multimodal context alignment video scene segmentation rule proposed above, the problem of audio and video semantic separation in the prior art is solved.

[0036] It should be understood that this application does not limit the values ​​of x and y, and those skilled in the art can make reasonable selections and settings according to the actual application scenario.

[0037] In one embodiment, the process of extracting frames from the video stream according to a preset first duration to generate video frames includes: determining the first duration based on the extraction target and purpose; and / or, determining the first duration by balancing information density and redundancy; and / or, segmenting the video stream based on the segment type and confirming the first duration of each segment.

[0038] Specifically, determining the first duration based on the frame extraction target and purpose includes: if it is used for quick preview of video content (such as thumbnail collections or key plot overviews), the core is to cover key scenes, and it is necessary to avoid missing key shot transitions and character action turning points (hereinafter referred to as key information nodes), so the first duration is determined based on key information nodes. Balancing information density and redundancy to determine the first duration includes: when the video stream is dynamic, if the first duration (e.g., 5s) is too long, it will cause large differences between adjacent frames and loss of key actions, so the first duration needs to be shortened (e.g., 0.3s-1s) to ensure the continuity of action trajectories; when the video stream is static / slow-motion video, if the first duration (e.g., 0.5s) is too short, it will generate a large number of duplicate frames, leading to a surge in storage / processing costs, so the first duration needs to be extended (e.g., 2s-3s) to retain only key information such as the points of change in the scene. The video stream is segmented based on the segment type, and the first duration of each segment is determined. When the video stream includes static and dynamic images, it is divided into multiple segments such as dynamic segments, static segments, and transition-intensive segments, and a different first duration is set for each segment (e.g., the first duration of the dynamic segment is 0.5s and the first duration of the static segment is 2s).

[0039] In one implementation, when determining the first duration, a balance between the "first duration" and the "actual screen information" should be considered to avoid the frame extraction results from the first duration being too singular and failing to meet subsequent usage requirements.

[0040] In one embodiment, fusing the target audio frame and the video frame to generate a video frame includes: packaging the target audio frame and video frame into a single file using the structure of audio and video formats (such as MP4, MKV), with the output file compatible with mainstream players / editing software and requiring no additional processing; or, directly binding the video frame and audio frame to generate independent audio and video frame pairs, with fine data association granularity, supporting quick retrieval of the corresponding audio frame by single video frame, suitable for scenarios such as monitoring frame-level evidence storage; or, if it is necessary to verify the alignment effect of the video frame and audio frame and retain the fusion basis, a synchronous verification file is generated synchronously during the fusion time, and the video frame and audio frame are fused to facilitate subsequent troubleshooting (such as tracing back the verification file to locate the cause if the audio and video suddenly become misaligned during playback).

[0041] In one embodiment, the process of clustering the video frames to generate a video segment sequence includes: obtaining the first na video frames and the first n+a video frames based on the nth video frame; obtaining the average type probability of each scene type in the target video segment based on a preset scene type; determining the maximum value of the average type probability and its corresponding scene type as the type probability value and scene type of the nth video frame; sequentially clustering video frames of the same scene type into video segments, and generating a video segment sequence based on the video segments; wherein the target video segment includes the nth video frame, the first na video frames, and the first n+a video frames, where n > a, and n and a are both positive integers.

[0042] Preferably, a is 1. For example... Figure 2 The diagram illustrates the rule for determining the target video segment when 'a' is 1. It should also be understood that this application does not limit the value of 'a', and those skilled in the art can make reasonable selections and settings based on actual circumstances.

[0043] Specifically, assuming a=1 and the number of video frames is 4, when obtaining the scene type and its probability value of the second video frame, the scene type and its probability value of the second video frame are determined based on each scene type and its probability value of the first and second video frames. Similarly, when obtaining the scene type and its probability value of the third video frame, the scene type and its probability value of the third video frame are determined based on each scene type and its probability value of the second and fourth video frames. This can be understood as determining the scene type and its probability value of any video frame based on sliding window statistical rules. Determining the scene type and its probability value of any video frame based on the rules described above improves the robustness of recognizing complex scene types.

[0044] It is important to understand that the scene type and its probability value of a video frame can be predicted based on a pre-generated target model. The pre-generated target model can be trained using a pre-acquired dataset and training set. The training and optimization of the target model are common techniques in the field of model training, and will not be elaborated here.

[0045] In one embodiment, the method further includes pre-generating a target model; the pre-generated target model specifically includes: First, prepare the data and clarify the scene type to be identified (such as "indoor", "outdoor" and "street"), and determine the corresponding video frames. Among them, the corresponding video frames can be reused from public scene datasets, or you can record videos and extract frames yourself (at least 500-1000 frames for each category, covering different lighting and angles to avoid insufficient model prediction ability). Secondly, label the corresponding video frames and the probability of labeling fuzzy frames (e.g., "the balcony looks like both indoors and outdoors, label it 0.7 for indoors and 0.3 for outdoors"). Finally, divide the set into a training set (for model learning), a validation set (for model optimization), and a test set (for model testing) in a 7:2:1 ratio. Then, build and train the model; the model can directly use mature image classification models (such as the lightweight MobileNet suitable for real-time scenarios, and the high-precision ResNet suitable for offline analysis), load the "basic capabilities" that the existing model already has on general images, and then adapt the model to suit the current application scenario; Next, the model is trained and optimized. Cross-entropy loss is used to calculate the model's prediction error, and the Adam optimizer is used to adjust parameters. The batch size is set according to the GPU memory (e.g., 16 or 32), and the model is trained for 20-50 epochs. If training is good but validation is poor, it indicates overfitting; add data or simplify the model. If both are poor, a more complex model is used. After training, misclassified video frames (e.g., misclassifying "balcony" as "living room") are used as supplementary data to complete the dataset and retrain.

[0046] Finally, the target model is generated and deployed. The model can be converted to deployment formats such as ONNX. The model will output the probability of each scene (e.g., "street 0.92, living room 0.05") by inputting pre-processed video frames. The scene type with the highest probability is selected. The training effect of the model can be monitored and continuously optimized.

[0047] In one embodiment, the scene types are defined as three types: a first scene type, a second scene type, and a third scene type. The average probability of the first scene type of the target video segment is 20%, the average probability of the second scene type is 30%, and the average probability of the third scene type is 50%. Then, the third scene type is determined to be the scene type of the nth video frame, and 50% is determined to be the type probability value of the nth video frame.

[0048] In one embodiment, obtaining the average type probability of each scene type of the target video segment based on a preset scene type includes: obtaining the type probability value of each video frame of the target video segment based on each scene type; and obtaining the average type probability of each scene type of the target video segment based on each scene type and the type probability value of each video frame.

[0049] Specifically, the target video segment includes video frames n, na, and (n+a)-th. The probability values ​​of the first, second, and third scene types for video frames n, na, and (n+a)-th are obtained respectively. Assuming the probability values ​​of the first, second, and third scene types for video frames n are 10%, 20%, and 20%, respectively... The probability values ​​of the first, second, and third scene types in the na-th video frame are 20%, 50%, and 30%, respectively, and the probability values ​​of the first, second, and third scene types in the (n+a)-th video frame are 30%, 20%, and 50%, respectively. Therefore, the average probability of the first scene type in the target video segment is (10%+20%+30%) / 3=20%. Similarly, the average probability of the second scene type in the target video segment is 30%, and the average probability of the third scene type in the target video segment is 50%.

[0050] In one implementation, the sequential clustering of video frames of the same scene type into video segments includes: obtaining the scene type of the nth video frame and the scene type of the (n-1)th video frame respectively; when the scene type of the nth video frame is consistent with the scene type of the (n-1)th video frame, clustering the video segments to which the nth video frame to the (n-1)th video frame belong.

[0051] Specifically, video frames of the same scene type are sequentially clustered based on their timestamps to form video segments, and the timestamps of the frames contained in the video frame are the timestamps of the video frame. When n is 1, if the scene type of the first video frame is the first scene type and the scene type of the second video frame is the first scene type, then a video segment is generated based on the first and second video frames, and the scene type of the video segment is the first scene type. If the scene type of the first video frame is the first scene type and the scene type of the second video frame is the second scene type, then a first video segment is generated based on the first video frame and a second video segment is generated based on the second video frame.

[0052] It should be understood that any video segment comprises several consecutive video frames, and all video frames have the same scene type; that is, if the scene type of the nth video frame is different from the scene type of the (n-1)th video frame, a new video segment is generated based on the nth video frame; if the scene type of the (n+1)th video frame is the same as the scene type of the nth video frame, the (n+1)th video frame is merged into the video segment to which the nth video frame belongs; if the scene type of the (n+1)th video frame is different from the scene type of the nth video frame, a new video segment is generated based on the (n+1)th frame, and the scene type of this video segment is the scene type of the (n+1)th video frame.

[0053] It is important to understand that if the scene type of the (n+1)th video frame is different from that of the nth video frame, even if the scene type of the (n+1)th video frame is the same as that of the (n-1)th video frame, the (n+1)th video frame will not be merged into the video segment to which the (n-1)th video frame belongs.

[0054] In one embodiment, generating a video segment sequence based on the video segments includes: obtaining the scene type of each video segment; clustering video segments with the same scene type to generate a video segment sequence; wherein, when there are multiple video segments, the timestamp of the previous video segment in the video segment sequence is earlier than the timestamp of the next video segment.

[0055] Specifically, the timestamp of a video segment is the timestamp of the last video frame contained in the video segment; in any video segment, the timestamp of the later video frame is later than the timestamp of the previous video frame; in any video segment sequence, the timestamp of the later video segment is later than the timestamp of the previous video frame.

[0056] In one embodiment, the method further includes: when there are multiple video segments, determining the order of the video segments based on the timestamps of the video segments; when the duration of the m-th video segment is not greater than a preset duration, obtaining a first scene type and a second scene type respectively; determining the scene type of the m-th video segment based on the first scene type and the second scene type; wherein the first scene type is the scene type of the (m-1)-th video segment, and the second scene type is the scene type of the (m+1)-th video segment.

[0057] Specifically, assuming the timestamp of the m-th video segment is 3 minutes and 15 seconds, the video segment whose timestamp is less than 3 minutes and 15 seconds and whose timestamp is closest to 3 minutes and 15 seconds is the (m-1)-th video segment; similarly, the video segment whose timestamp is greater than 3 minutes and 15 seconds and whose timestamp is closest to 3 minutes and 15 seconds is the (m+1)-th video segment.

[0058] In one embodiment, determining the scene type of the m-th video segment based on the first scene type and the second scene type includes: when the first scene type and the second scene type are inconsistent, determining the type probability value of the m-th video segment corresponding to the first scene type as a first probability value, determining the type probability value of the m-th video segment corresponding to the second scene type as a second probability value, determining the type probability value of the first scene type as a third probability value, and determining the type probability value of the second scene type as a fourth probability value; when the difference between the first probability value and the third probability value is greater than the difference between the second probability value and the fourth probability value, determining the second scene type as the scene type of the m-th video segment; when the difference between the first probability value and the third probability value is less than the difference between the second probability value and the fourth probability value, determining the first scene type as the scene type of the m-th video segment.

[0059] Specifically, the scene type of the m-th video segment is the second scene type. The difference between the type probability value of the preceding video segment (such as the first scene type) and the type probability value of the same type of the m-th video segment (i.e., the type probability value of the first scene type) is diff_A; the difference between the type probability value of the following video segment (such as the third scene type) and the type probability value of the same type of the m-th video segment (i.e., the type probability value of the third scene type) is diff_B. When diff_A is less than diff_B, the m-th video segment is merged into the preceding video segment; otherwise, it is merged into the following video segment.

[0060] Based on the video segment sequence generation method described above, a video segment sequence corresponding to each scene type can be generated for the target video. Each video segment sequence contains several video segments, and these video segments are arranged sequentially based on their timestamps to determine the coherence of the video segments under each scene type. It should be understood that the video frame mentioned in this application refers to a multimodal data unit that includes a picture frame and its corresponding target audio frame after fusion.

[0061] Based on the same inventive concept, this application also provides a video segment sequence generation device, the structure of which is as follows: Figure 2 As shown, the video segment sequence generation device includes at least: The parsing unit is used to parse the target video to extract the video stream and audio stream in response to a video segment sequence generation request; The frame extraction unit is used to extract frames from the video stream according to a preset first duration to generate video frames and their timestamps, and to perform aligned frame extraction from the audio stream based on the timestamps and a preset second duration to generate target audio frames. A fusion unit is used to fuse the target audio frame and the picture frame to generate a video frame, and to cluster the video frames to generate a video segment sequence.

[0062] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the video segment sequence generation method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0063] The electronic device can be a user device, or a device formed by integrating user devices and network devices through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0064] Figure 3 The diagram illustrates the structure of an apparatus suitable for implementing the methods and / or technical solutions in the embodiments of this application. The apparatus 300 includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in a Read Only Memory (ROM) 302 or a program loaded from a storage portion 308 into a Random Access Memory (RAM) 303. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0065] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 308 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet.

[0066] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 301, it performs the functions defined in the methods of this application.

[0067] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0068] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0069] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0070] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0071] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0072] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0073] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0077] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0079] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for generating video segment sequences, characterized in that, The method includes: In response to a video segment sequence generation request, the target video is parsed to extract the video and audio streams; The video stream is frame-by-frame extracted according to a preset first duration to generate video frames and their timestamps. Based on the timestamp and the preset second duration, the audio stream is aligned and frames are extracted to generate the target audio frame; The target audio frame and the picture frame are merged to generate a video frame, and the video frames are clustered to generate a video segment sequence.

2. The video segment sequence generation method according to claim 1, characterized in that, The clustering of the video frames to generate video segment sequences includes: Based on the nth video frame, obtain the first na video frames and the first n+a video frames respectively; Based on preset scene types, obtain the type probability value of each scene type in the target video segment; The maximum value of the type probability value and its corresponding scene type are determined as the type probability value and scene type of the nth video frame; Video frames of the same scene type are sequentially clustered into video segments, and a video segment sequence is generated based on the video segments. The target video segment includes the nth video frame, the previous na video frames, and the previous n+a video frames, where n > a, and n and a are both positive integers.

3. The video segment sequence generation method according to claim 2, characterized in that, The sequential clustering of video frames of the same scene type constitutes video segments, including: Obtain the scene type of the nth video frame and the scene type of the (n-1)th video frame respectively; When the scene type of the nth video frame is the same as that of the (n-1)th video frame, the video segments to which the nth to (n-1)th video frames belong are clustered.

4. The video segment sequence generation method according to claim 3, characterized in that, The sequential clustering of video frames of the same scene type into video segments further includes: When the scene type of the nth video frame is inconsistent with the scene type of the (n-1)th video frame, a new video segment is generated based on the nth video frame. The scene type of the nth video frame is determined to be the scene type of the video segment.

5. The video segment sequence generation method according to any one of claims 1 to 4, characterized in that, The method further includes: When there are multiple video segments, the order of the video segments is determined based on their timestamps. If the duration of the m-th video segment is not greater than the preset duration, then the first scene type and the second scene type are obtained respectively. Based on the first scene type and the second scene type, determine the scene type of the m-th video segment; Wherein, the first scene type is the scene type of the (m-1)th video segment, and the second scene type is the scene type of the (m+1)th video segment.

6. The video segment sequence generation method according to claim 5, characterized in that, Determining the scene type of the m-th video segment based on the first scene type and the second scene type includes: When the first scene type and the second scene type are inconsistent, the type probability value of the first scene type corresponding to the m-th video segment is determined to be a first probability value, the type probability value of the second scene type corresponding to the m-th video segment is determined to be a second probability value, the type probability value of the first scene type is determined to be a third probability value, and the type probability value of the second scene type is determined to be a fourth probability value. When the difference between the first probability value and the third probability value is greater than the difference between the second probability value and the fourth probability value, the second scene type is determined to be the scene type of the m-th video segment; When the difference between the first probability value and the third probability value is less than the difference between the second probability value and the fourth probability value, the first scene type is determined to be the scene type of the m-th video segment.

7. The video segment sequence generation method according to claim 2, 3, 4 or 6, characterized in that, The step of obtaining the type probability value of each scene type in the target video segment based on preset scene types includes: Based on each scene type, obtain the type probability value of each video frame of the target video segment; Based on each scene type and the type probability value of each video frame, obtain the type probability value of each scene type in the target video segment.

8. The video segment sequence generation method according to claim 1, 2, 3, 4 or 6, characterized in that, Based on the timestamp and a preset second duration, the audio stream is aligned and frames are extracted to generate a target audio frame, including: Based on the timestamp, the audio frame with a second duration x times before the timestamp is determined as the first audio frame, and the audio frame with a second duration y times after the timestamp is determined as the second audio frame. A target audio frame is generated based on the first audio frame and the second audio frame.

9. The video segment sequence generation method according to claim 1, 2, 3, 4 or 6, characterized in that, The process of generating a video segment sequence based on the video segment includes: Obtain the scene type for each video segment; Cluster video segments with the same scene type to generate a video segment sequence; When there are multiple video segments, the timestamp of the first video segment in the video segment sequence is earlier than the timestamp of the second video segment.

10. A video segment sequence generation device, characterized in that, The device includes: The parsing unit is used to parse the target video to extract the video stream and audio stream in response to a video segment sequence generation request; The frame extraction unit is used to extract frames from the video stream according to a preset first duration to generate video frames and their timestamps, and to perform aligned frame extraction from the audio stream based on the timestamps and a preset second duration to generate target audio frames. A fusion unit is used to fuse the target audio frame and the picture frame to generate a video frame, and to cluster the video frames to generate a video segment sequence.

11. An electronic device, the electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; characterized in that, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the video segment sequence generation method according to any one of claims 1 to 9.

12. A computer-readable medium having computer program instructions stored thereon, characterized in that, The computer program instructions can be executed by a processor to implement the video segment sequence generation method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video segment sequence generation method as described in any one of claims 1 to 9.

Images