Video processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a video processing method and device, and relates to the technical field of computers, in particular to the technical fields of large models, deep learning, image processing and the like. According to the specific implementation scheme, shot segmentation is carried out on an obtained to-be-processed video to obtain a plurality of video shot segments; extracting video information of each video shot clip in the plurality of video shot clips; obtaining a semantic time period based on the video information; and obtaining a semantic shot fragment based on the semantic time period and the plurality of video shot fragments.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, specifically to technical fields such as large models, deep learning, and image processing, and in particular to a video processing method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Currently, watching short videos has become a way for people to relax in their daily lives, and various types of pan-entertainment and comedy content are emerging in an endless stream. With the explosion of the short drama market in recent years, users watching short dramas on short video platforms has become a new form of entertainment. However, short dramas or film and television dramas have strong coherence, and users cannot complete content consumption through fragmented time. This model not only reduces the user's experience of consuming content, but may also cause users to lose interest in the platform, thereby reducing the platform's activity and user stickiness. Summary of the invention

[0003] The present disclosure provides a video processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect, a video processing method is provided, the method comprising: performing shot segmentation on an acquired video to be processed to obtain a plurality of video shot segments; extracting video information of each of the plurality of video shot segments; obtaining a semantic time segment based on the video information; and obtaining a semantic shot segment based on the semantic time segment and the plurality of video shot segments.

[0005] According to a second aspect, a video processing device is provided, which includes: a shot obtaining unit, configured to perform shot segmentation on the acquired video to be processed to obtain multiple video shot segments; an extraction unit, configured to extract video information of each video shot segment in the multiple video shot segments; a time obtaining unit, configured to obtain a semantic time period based on the video information; and a semantic obtaining unit, configured to obtain a semantic shot segment based on the semantic time period and the multiple video shot segments.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0009] The video processing method and device provided by the embodiment of the present disclosure first divide the acquired video to be processed into shots to obtain multiple video shot segments; secondly, extract the video information of each video shot segment in the multiple video shot segments; thirdly, obtain the semantic time segment based on the video information; finally, obtain the semantic shot segment based on the semantic time segment and the multiple video shot segments. Thus, after dividing the video to be processed into shots, multiple video shot segments are spliced ​​by semantic time segments to obtain semantic shot segments, which provides reliable support for obtaining semantic shot segments and realizes the automatic extraction effect of semantic shot segments; the obtained semantic shot segments can be used to annotate the video to be processed, which provides a reliable basis for video annotation.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a flow chart of an embodiment of a video processing method according to the present disclosure;

[0013] Figure 2 is a flowchart of another embodiment of the video processing method according to the present disclosure;

[0014] Figure 3 It is a structural schematic diagram of the present disclosure showing display points on the preview timeline;

[0015] Figure 4 is a structural schematic diagram of an embodiment of a video processing device according to the present disclosure;

[0016] Figure 5 The block diagram is a block diagram of an electronic device for implementing the video processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] In traditional technology, the annotation of key plots and highlights in videos is generally done manually by platform staff using tools provided by the platform. Since it only supports platform personnel to annotate content, the coverage is limited and cannot meet the annotation needs of video creators. It is also time-consuming and has low overall production efficiency.

[0019] In view of the defects in the traditional technology, the present invention proposes a video processing method, which can automatically extract the annotation information of the video to be processed, and provide reliable support for the content annotation of the video to be processed. Figure 1 A process 100 according to an embodiment of a video processing method of the present disclosure is shown. The video processing method comprises the following steps:

[0020] Step 101, segmenting the acquired video to be processed into shots to obtain a plurality of video shot segments.

[0021] In this embodiment, the video to be processed can be a whole video or multiple segmented videos (such as a short play). The video to be processed can include video content and can also include: video cover image. The video shot segment is a video segment including a shot in the video to be processed, wherein a shot can be the video content of a subject (person, object, etc.) in the video to be processed, or a shot can also be the video content of a background in the video to be processed.

[0022] In this embodiment, the execution subject on which the video processing method runs can be a cloud or a client. After the execution subject obtains the video to be processed, it uses the physical shot segmentation rule to segment the video to be processed to obtain multiple video shot segments. Among them, the physical shot segmentation rule can be a rule that limits the shot time. For example, the video content in the video to be processed is information about the changes in the subject shot according to the shot time. By segmenting the video to be processed according to the shot time rule, each video shot segment can have corresponding shot information. For this purpose, each video shot segment can be a video with one shot for 1 minute or 30 seconds. Alternatively, the physical shot segmentation rule can be a shot rule. For example, each shot switch of the video to be processed can correspond to a video segment. For this purpose, each video shot segment can be a video including a subject or a background.

[0023] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the videos to be processed are carried out after authorization and in compliance with relevant laws and regulations. The information related to the user in the videos to be processed is information obtained after the user's permission, and the video processing related to the user is carried out under confidentiality conditions.

[0024] Step 102: extract video information of each video shot segment among a plurality of video shot segments.

[0025] In this embodiment, the video information may be one or more of image information, text information, and voice information in a video shot segment, wherein the image information in each video shot segment may be directly identified through an image recognition algorithm and an image recognition model, the text information in each video shot segment may be directly identified through a text recognition algorithm and a text recognition model, and the voice information in each video shot segment may be directly identified through a voice recognition algorithm and a voice recognition model.

[0026] Optionally, the above step 102 includes: extracting ASR (Automatic Speech Recognition, speech recognition technology) information and OCR (Optical Character Recognition, optical character recognition) information for each video shot segment, and using the obtained ASR information and OCR information as video information.

[0027] Step 103: Obtain a semantic time period based on the video information.

[0028] In this embodiment, the semantic time period is the time that reflects the integrity of entities such as events, stories, plots, and scenes in the video to be processed, and the semantic time period is the overall development time of an entity. The above step 103 includes: determining the time period of each video lens segment; based on the video information of each video lens segment, determining the video lens segments that include the same storyline; merging the video lens segments that include the same storyline, and using the time period of the merged video lens segments as the semantic time period.

[0029] Optionally, the above step 103 may also include: determining multiple storylines in the video to be processed; classifying the video information belonging to each storyline into each category; determining and merging the video footage segments belonging to each category, and using the time period of the merged video footage segments under each category as the semantic time period.

[0030] Step 104: obtaining semantic shot segments based on the semantic time period and the plurality of video shot segments.

[0031] In this embodiment, the semantic shot fragment is a video fragment that reflects the integrity of the event, story and scene in the video to be processed. The semantic shot fragment can be a sequential combination of video shot fragments. The semantic shot fragment can also be a video shot fragment. When necessary, the semantic shot fragment can also be a video fragment extracted from multiple different video shot fragments in succession. For example, the multiple video shot fragments include: 1, 2, 3, 4, and 5 segments, and the semantic shot fragment includes a first shot fragment and a second shot fragment, wherein the first shot fragment is obtained by combining video shot fragments 1 and 3, and the second shot fragment is obtained by combining video shot fragments 4 and 5.

[0032] In this embodiment, a new timeline is generated for the semantic shot fragments. Each semantic time period on the timeline is the time corresponding to the time of the corresponding semantic shot. There can be one semantic time period and one semantic shot fragment, or there can be multiple semantic time periods and multiple semantic shot fragments. When a semantic time period belongs to a video shot fragment, the video shot fragment is used as a semantic fragment; when a semantic time period spans multiple video shot fragments, the multiple video shot fragments are merged to obtain a semantic shot fragment.

[0033] The video processing method disclosed herein solves existing problems and meets market demand through integrated design tools and rich interactive support: it can provide creators with a visual interface and intelligent design tools, implement actual application scenarios, and help creators quickly identify key content and annotated moments in the video without the need for experience with large models, thereby reducing time and technical costs.

[0034] The video processing method provided by the embodiment of the present disclosure first divides the acquired video to be processed into shots to obtain multiple video shot segments; secondly, extracts the video information of each video shot segment in the multiple video shot segments; thirdly, obtains the semantic time segment based on the video information; finally, obtains the semantic shot segment based on the semantic time segment and the multiple video shot segments. Thus, after dividing the video to be processed into shots, the multiple video shot segments are spliced ​​by the semantic time segment to obtain the semantic shot segment, which provides reliable support for obtaining the semantic shot segment and realizes the automatic extraction effect of the semantic shot segment; the obtained semantic shot segment can be used to annotate the video to be processed, which provides a reliable basis for video annotation.

[0035] In some optional implementations of the present disclosure, the above-mentioned segmentation of the acquired video to be processed to obtain multiple video shot segments includes: acquiring the video to be processed; inputting the video to be processed into a pre-trained segmentation model to obtain multiple video shot segments output by the segmentation model.

[0036] In this optional implementation, the video to be processed can be a whole video or multiple segmented videos (such as a short play).

[0037] In this optional implementation, when the video to be processed is multiple short dramas, the segmentation model can be a Transnetv2 model based on 3D convolution. Through this segmentation model, all the short drama videos in the video to be processed can be automatically physically segmented, so that the short dramas can be divided into different video shot segments.

[0038] The method for obtaining multiple video shot segments provided by this optional implementation method obtains multiple video shot segments by inputting the acquired video to be processed into a pre-trained segmentation model to obtain multiple video shot segments output by the segmentation model, thereby providing a reliable implementation method for obtaining multiple video shot segments.

[0039] Optionally, the above-mentioned shot segmentation of the acquired video to be processed to obtain multiple video shot segments includes: acquiring the video to be processed; identifying the subject and background of each image frame in the video to be processed; and in response to changes in the subject or background of adjacent image frames, dividing the image frames in the adjacent image frames into different shot segments.

[0040] In some optional implementations of the present disclosure, the above-mentioned video information includes: text information and voice information; obtaining the semantic time period based on the video information includes: calculating the editing distance between any two adjacent video shot segments in a plurality of video shot segments based on the text information of the two video shot segments; in response to the editing distance being less than a distance threshold, merging the time periods of the two video shot segments to obtain an intermediate time period; detecting whether the intermediate time period is within the voice time period corresponding to the two video shot segments, or intersecting with the voice time period corresponding to the two video shot segments; in response to detecting that the intermediate time period is within the voice time period corresponding to the two video shot segments, or that the intermediate time period intersects with the voice time period corresponding to the two video shot segments, expanding the intermediate time period based on the voice time period to obtain the semantic time period.

[0041] In this optional implementation, the edit distance is a metric used to measure the difference between two strings. It defines the similarity between two strings by calculating the minimum number of single-character edit operations required to convert from one string to another. The distance threshold can be set based on development requirements, for example, the distance threshold is 1.

[0042] In this optional implementation, the middle time period is the time period after the time periods of two adjacent video lens segments are merged. For example, if the time periods of two adjacent video lens segments are a to b and b to c respectively, the middle time period is a to c.

[0043] In this optional implementation, for text information, the edit distance of adjacent video footage segments is calculated. When the edit distance is less than the distance threshold, the time segments of adjacent video footage segments are merged to obtain the middle time segment, so as to achieve the purpose of expanding the time segment. Then, based on the middle time segment axis, it is determined whether there is an overlap with the speech time segment. For the overlapping part, the middle time segment is expanded as the semantic time segment.

[0044] In this optional implementation, the semantic time period is a time period formed by the time points of the video to be processed. For example, when the middle time period is 4-6s and the voice time period is 3.5-6.5s, it is determined that the middle time period is within the voice time period corresponding to the two video lens segments, and the semantic time period is 3.5-6.5s. When the middle time period is 3-6s and the voice time period is 3.5-6.5s, it is determined that the middle time period overlaps with the voice time period corresponding to the two video lens segments, and the semantic time period is 3-6.5s.

[0045] The method for obtaining the intention recognition result of user interaction information provided by this optional implementation method determines the recognition model group and the priority of each recognition model group in the recognition model set based on the information to be processed; controls each recognition model group to work according to its own priority, and dynamically selects the recognition model based on the information to be processed, thereby improving the accuracy of the intention recognition result of user interaction information.

[0046] Optionally, obtaining the semantic time period based on the video information also includes: in response to the voice information being empty, calculating the edit distance between any two adjacent video shot segments among the multiple video shot segments based on the text information of the two video shot segments; in response to the edit distance being less than a distance threshold, merging the time periods of the two video shot segments to obtain the semantic time period.

[0047] Optionally, obtaining the semantic time period based on the video information further includes: in response to the edit distance being greater than or equal to a distance threshold, taking each of the two video shot segments as the semantic time period.

[0048] Optionally, based on the video information, obtaining the semantic time period also includes: in response to the text information being empty, obtaining voice information of any two adjacent video shot clips among multiple video shot clips; in response to the voice information of the two video shot clips belonging to the same subject, merging the time periods of the two video shot clips to obtain the semantic time period.

[0049] In some optional implementations of the present disclosure, obtaining the semantic time period based on the video information further includes: in response to detecting that the voice time periods corresponding to the two video shot segments are within the middle time period, using the middle time period as the semantic time period.

[0050] The method for obtaining the semantic time period provided by this optional implementation uses the middle time period as the semantic time period when the speech time periods corresponding to the two video shot segments are within the middle time period, thereby providing another reliable implementation method for obtaining the semantic time period.

[0051] In some optional implementations of the present disclosure, the above-mentioned obtaining of semantic shot segments based on multiple semantic time periods and multiple video shot segments includes: detecting whether the semantic time period spans across multiple video shot segments; in response to detecting that the semantic time period spans across multiple video shot segments, merging the multiple video shot segments spanned by the semantic time period to obtain the semantic shot segments of the semantic time period.

[0052] In this optional implementation, it is determined whether the semantic time period spans across multiple video lens segments. If it spans across multiple video lens segments, the multiple video lens segments are finally merged to generate a semantic lens segment. For example, the multiple video lens segments are 1-3 and 3-5, but the video lens segment corresponding to the semantic time period is 2-4. At this time, the semantic time period spans multiple video lens segments, so the merged reasonable shots are 1-5 as the semantic lens segment.

[0053] The method for obtaining semantic shot fragments provided by this optional implementation mode, when the semantic time period spans across multiple video shot fragments, merges the multiple video shot fragments spanning the semantic time period, so that the obtained semantic shot fragments can maximize the number of videos belonging to the same storyline, thereby improving the reliability of obtaining the semantic shot fragments.

[0054] In some optional implementations of the present disclosure, the above-mentioned obtaining of semantic shot segments based on the semantic time period and multiple video shot segments also includes: in response to detecting that the semantic time period does not span multiple video shot segments, merging the video shot segments corresponding to the semantic time period to obtain the semantic shot segments of the semantic time period.

[0055] The method for obtaining semantic shot segments provided by this optional implementation mode combines video shot segments corresponding to the semantic time period to obtain semantic shot segments when the semantic time does not span multiple video shot segments, thereby ensuring the overall effect of the semantic shot segments.

[0056] Optionally, obtaining the semantic shot segments based on the semantic time segment and the multiple video shot segments further comprises: in response to detecting that the semantic time segment does not span across the multiple video shot segments, taking all the video shot segments corresponding to the semantic time segment as the semantic shot segments.

[0057] Figure 2 A process 200 according to an embodiment of a video processing method of the present disclosure is shown. The video processing method comprises the following steps:

[0058] Step 201, segmenting the acquired video to be processed into shots to obtain a plurality of video shot segments.

[0059] Step 202: extract video information of each video shot segment among a plurality of video shot segments.

[0060] Step 203: Obtain a semantic time period based on the video information.

[0061] Step 204: obtaining semantic shot segments based on the semantic time period and the plurality of video shot segments.

[0062] In this embodiment, it should be understood that the operations and features in the above steps 201 to 204 correspond to the operations and features in steps 101 to 104, respectively. Therefore, the above descriptions of the operations and features in steps 101 to 104 are also applicable to steps 201 to 204 and will not be repeated here.

[0063] Step 205: Obtain annotation information of the video to be processed based on the semantic shot segments.

[0064] In this embodiment, the annotation information is to classify the semantic lens segments in the processed video, and determine the information to be displayed to the user based on the classification type. The annotation information includes: the classification type of each semantic lens segment and the display content corresponding to each classification type, wherein the classification type can be set based on the display needs of the user, for example, the classification type includes: at least one subject-specific action type, when the semantic lens segment is classified into the subject-specific action type, the display content corresponding to the classification type can be directly displayed on the player that plays the semantic lens segment. The display content can be directly information such as text, images, and symbols.

[0065] It should be noted that the displayed content is the content that can be displayed on the preview timeline of the video to be processed. When the semantic shot segment is a video segment obtained by sequentially merging at least one video shot segment, the displayed content can be directly displayed at the preview timeline position corresponding to the video shot segment that is the earliest in time. When the semantic shot segment is not a video segment obtained by sequentially merging at least one video shot segment, for example, a semantic shot segment is obtained by selecting a video shot segment at intervals from multiple video shot segments and merging them, the displayed content can be at the preview timeline position corresponding to all the selected video shot segments.

[0066] The video processing method provided by the embodiment of the present disclosure first divides the acquired video to be processed into shots to obtain multiple video shot segments; secondly, extracts the video information of each video shot segment in the multiple video shot segments; thirdly, obtains the semantic time segment based on the video information; and then, obtains the semantic shot segment based on the semantic time segment and the multiple video shot segments; finally, obtains the annotation information of the video to be processed based on the semantic shot segment. Thus, after dividing the video to be processed into shots, multiple video shot segments are spliced ​​by semantic time segments to obtain semantic shot segments, which provides reliable support for obtaining semantic shot segments; based on the semantic shot segments, the annotation information of the video to be processed is obtained, and the marking of the video can be automatically realized, which realizes the automation effect of video annotation and improves the efficiency of video content annotation.

[0067] In some embodiments of the present disclosure, the above-mentioned video processing method further includes: in response to receiving display information of the video to be processed, displaying a display point corresponding to the annotation information on the preview timeline of the video to be processed.

[0068] In this embodiment, the display information is information indicating that the video to be processed is about to be or has been displayed in the player, and the display information may be user operation information on the video to be processed, such as single-click or double-click.

[0069] In this embodiment, the preview timeline of the video to be processed is the timeline when the video to be processed is displayed in the player. A progress bar is set on the preview timeline. The progress bar is usually used to display the progress of video playback, helping users to intuitively understand the playback status of the video, and can provide visual feedback so that users can clearly see the playback position and remaining time of the video. Figure 3 As shown, the video to be processed is being played on the player, and the overall duration of the video to be processed is 2 minutes and 31 seconds. Two display points D are displayed on the preview timeline Y on the player. The two display points are extracted by the annotation information obtained by the video processing method of the present invention, and the video segment between the two display points is a semantic lens fragment. By dragging the progress bar T on the preview timeline Y to the display point D, the player can directly play the semantic lens fragment corresponding to the display point D.

[0070] In this embodiment, firstly, it is determined whether the display information of the video to be processed is received; if the display information of the video to be processed is received, the display point corresponding to the annotation information is displayed on the preview time axis of the video to be processed. Otherwise, the process directly exits.

[0071] In this embodiment, the display point is the content that can be displayed on the preview timeline of the video to be processed. When the semantic shot segment is a video segment obtained by sequentially merging at least one video shot segment, the display point can be directly displayed at the preview timeline position corresponding to the video shot segment that is the earliest in time. When the semantic shot segment is not a video segment obtained by sequentially merging at least one video shot segment, for example, a video shot segment is selected every other video shot segment to merge to obtain the semantic shot segment, the display point can be at the preview timeline position corresponding to all the selected video shot segments.

[0072] The video processing method provided in this embodiment, after receiving the display information of the video to be processed, displays the display points corresponding to the annotation information on the preview timeline of the video to be processed, so that the user can conveniently and quickly watch the highlight content and key plots in the video to be processed through the display points, thereby improving the user experience and the efficiency of video annotation.

[0073] In some optional implementations of the present disclosure, the above-mentioned annotation information includes: annotation time and annotation type. Based on the semantic shot fragments, obtaining the annotation information of the video to be processed includes: inputting the semantic shot fragments into a pre-trained multimodal large model, and obtaining the annotation results of each semantic shot fragment output by the multimodal large model; in response to the annotation result being required to be labeled, obtaining the annotation type from the annotation result; and using the initial time of the semantic shot fragment as the annotation time.

[0074] In this optional implementation, the marked time is the time when the marked information needs to be displayed, and the marked time can be used to determine the dot position on the preview timeline of the video to be processed. The annotation type is a classification of the current semantic lens segment. When the display point corresponding to the annotation information is displayed on the player, the marked time and the annotation type can be directly displayed on the preview timeline, or the marked time and the annotation type can be not displayed on the preview timeline.

[0075] In this optional implementation, the marked time point is the time point of the shot. The marked type will be stored in the material library. The page that the user actually sees does not display the highlight type. Figure 3 The display point D shown is shown. Based on the identified annotation type and annotation time point, when the user previews the timeline for dot display, the user can be guided to quickly complete the viewing of the annotated content.

[0076] In this optional implementation, the multimodal large model is used to classify semantic lens fragments. The multimodal large model can be a binary classification model or a multi-classification model. When the multimodal large model is a binary classification model, the annotation results given by the multimodal large model can be required or not required. When annotation is required, there is only one annotation type. When the multimodal large model is a multi-classification model, the annotation results given by the multimodal large model can be required or not required. When annotation is required, there can be multiple annotation types in the annotation results.

[0077] In this optional implementation, the multimodal big model is a pre-trained big model, which is used to implement classification of multiple modalities. By inputting semantic lens fragments into the multimodal big model, the labeling results output by the multimodal big model can be obtained. The labeling results are used to characterize whether the semantic lens fragments need to be labeled and the type of labeling.

[0078] In this optional implementation, for some annotation information that marks dangerous actions, the annotation types may include: physical conflict, acts of harming others, controlling others with force, kneeling, etc., more than a dozen first-level highlights and more than thirty second-level types.

[0079] In this optional implementation, the video to be processed may be a plurality of short dramas, for example, 100 episodes of short dramas, and the semantic shot segments are video segments with story development plots separated from the plurality of short dramas, which are identified using a multimodal large model and output labeling information of whether there are highlights and their highlight types.

[0080] The method for obtaining the annotation information of the video to be processed provided by this optional implementation analyzes the semantic shot fragments through a multimodal large model to obtain the annotation results of the semantic shot fragments, and obtains the annotation information of the video to be processed based on the annotation results, thereby improving the accuracy of obtaining the annotation information.

[0081] Optionally, the above-mentioned obtaining of annotation information of the video to be processed based on the semantic shot fragment includes: identifying the image in the semantic shot fragment to obtain action information of the subject in the image; based on the action information, determining whether the action of the subject belongs to an action in a preset annotation type, and if it belongs to an action in the preset annotation type, obtaining the annotation type of the semantic shot fragment.

[0082] In one embodiment of the present disclosure, the video processing method further includes: detecting whether there are two semantic shot segments whose difference in marking time is less than a time threshold; in response to detecting two semantic shot segments whose difference in marking time is less than the time threshold, detecting whether the marking types of the two semantic shot segments are the same; in response to detecting that the marking types of the two semantic shot segments are the same, merging the marking types of the two semantic shot segments and aggregating the marking times of the two semantic shot segments.

[0083] In this optional implementation, the time threshold can be set based on development needs, for example, the time threshold is 20s.

[0084] In this optional implementation, aggregating the annotated time of the two semantic shot segments refers to merging the annotated time of the two semantic shot segments together and using the earliest time. For example, if the annotated times of the two semantic shot segments are 01:00 and 01:30 respectively, 01:00 is used as the annotated time after the two semantic shot segments are aggregated.

[0085] The video processing method provided by the embodiment of the present disclosure detects the annotation types of two semantic shot segments when the difference between the annotation times of the two semantic shot segments is less than a time threshold, and merges the two semantic shot segments when the annotation types of the two semantic shot segments are the same, thereby improving the neatness of the semantic shots.

[0086] Optionally, the above-mentioned video processing method also includes: in response to detecting that the annotation types of two semantic shot segments are the same, detecting whether the difference between the annotation times of the two semantic shot segments is less than a time threshold; in response to the difference between the annotation times of the two semantic shot segments being less than the time threshold, aggregating the annotation times of the two semantic shot segments.

[0087] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a video processing device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0088] like Figure 4 As shown, the video processing device 400 provided in this embodiment includes: a shot obtaining unit 401, an extraction unit 402, a time obtaining unit 403, and a semantic obtaining unit 404. Among them, the above-mentioned receiving unit 401 can be configured to receive user interaction information sent by the front end. The above-mentioned shot obtaining unit 401 can be configured to perform shot segmentation on the acquired video to be processed to obtain multiple video shot segments. The above-mentioned extraction unit 402 can be configured to extract video information of each video shot segment in the multiple video shot segments. The above-mentioned time obtaining unit 403 can be configured to obtain a semantic time period based on the video information. The above-mentioned semantic obtaining unit 404 can be configured to obtain a semantic shot segment based on the semantic time period and multiple video shot segments.

[0089] In this embodiment, the specific processing of the lens obtaining unit 401, the extraction unit 402, the time obtaining unit 403, and the semantics obtaining unit 404 and the technical effects thereof can be referred to in detail. Figure 1The relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment are not repeated here.

[0090] In some optional implementations of this embodiment, the above-mentioned shot obtaining unit 401 is configured to: obtain a video to be processed; input the video to be processed into a pre-trained segmentation model to obtain a plurality of video shot segments output by the segmentation model.

[0091] In some optional implementations of the present embodiment, the video information includes: text information and voice information; the time obtaining unit 403 is configured to: calculate the editing distance between any two adjacent video shot segments in a plurality of video shot segments based on the text information of the two video shot segments; in response to the editing distance being less than a distance threshold, merge the time periods of the two video shot segments to obtain an intermediate time period; detect whether the intermediate time period is within the voice time period corresponding to the two video shot segments, or overlaps with the voice time period corresponding to the two video shot segments; in response to detecting that the intermediate time period is within the voice time period corresponding to the two video shot segments, or that the intermediate time period overlaps with the voice time period corresponding to the two video shot segments, expand the intermediate time period based on the voice time period to obtain a semantic time period.

[0092] In some optional implementations of this embodiment, the time obtaining unit 403 is further configured to: in response to detecting that the speech time periods corresponding to the two video shot segments are within the middle time period, use the middle time period as the semantic time period.

[0093] In some optional implementations of this embodiment, the semantic obtaining unit 404 is configured to: detect whether the semantic time period spans across multiple video shot segments; in response to detecting that the semantic time period spans across multiple video shot segments, merge the multiple video shot segments spanned by the semantic time period to obtain the semantic shot segment of the semantic time period.

[0094] In some optional implementations of this embodiment, the semantic obtaining unit 404 is further configured to: in response to detecting that the semantic time period does not span multiple video shot segments, merge the video shot segments corresponding to the semantic time period to obtain the semantic shot segments of the semantic time period.

[0095] In some optional implementations of this embodiment, the apparatus 400 further includes: a labeling obtaining unit (not shown in the figure), and the labeling obtaining unit is configured to obtain labeling information of the video to be processed based on the semantic shot fragments.

[0096] In some optional implementations of this embodiment, the above-mentioned device 400 also includes: a dot-marking unit (not shown in the figure), which is configured to display display points corresponding to the annotation information on the preview timeline of the video to be processed in response to receiving display information of the video to be processed.

[0097] In some optional implementations of the present embodiment, the above-mentioned annotation information includes: annotation time and annotation type, and the above-mentioned annotation obtaining unit is configured to: input the semantic lens fragment into a pre-trained multimodal large model to obtain the annotation results of each semantic lens fragment output by the multimodal large model; in response to the annotation result being that annotation is required, obtain the annotation type from the annotation result; and use the initial time of the semantic lens fragment as the annotation time.

[0098] In some optional implementations of the present embodiment, the above-mentioned device also includes: an aggregation unit (not shown in the figure), which is configured to: detect whether there are two semantic shot segments whose difference in marking time is less than a time threshold; in response to detecting two semantic shot segments whose difference in marking time is less than the time threshold, detect whether the marking types of the two semantic shot segments are the same; in response to detecting that the marking types of the two semantic shot segments are the same, merge the marking types of the two semantic shot segments, and aggregate the marking times of the two semantic shot segments.

[0099] The video processing method provided by the embodiment of the present disclosure is as follows: first, the lens obtaining unit 401 performs lens segmentation on the acquired video to be processed to obtain multiple video lens segments; secondly, the extraction unit 402 extracts the video information of each video lens segment in the multiple video lens segments; thirdly, the time obtaining unit 403 obtains the semantic time period based on the video information; finally, the semantic obtaining unit 404 obtains the semantic lens segment based on the semantic time period and the multiple video lens segments. Thus, after the video to be processed is segmented, the multiple video lens segments are spliced ​​by the semantic time period to obtain the semantic lens segment, which provides reliable support for obtaining the semantic lens segment and realizes the automatic extraction effect of the semantic lens segment; the obtained semantic lens segment can be used to annotate the video to be processed, which provides a reliable basis for the annotation of the video.

[0100] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0101] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0102] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0103] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0104] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0105] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as video processing methods. For example, in some embodiments, the video processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the video processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the video processing method in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable video processing device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0108] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0111] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video processing method, the method comprising: Segment the acquired video to be processed into shots to obtain multiple video shot segments; Extracting video information of each video shot segment from the plurality of video shot segments; Based on the video information, obtaining a semantic time segment; A semantic shot segment is obtained based on the semantic time period and the multiple video shot segments.

2. The method according to claim 1, wherein: The step of segmenting the acquired video to be processed to obtain a plurality of video segment segments includes: Get the video to be processed; The video to be processed is input into a pre-trained segmentation model to obtain a plurality of video shot segments output by the segmentation model.

3. The method according to claim 1, wherein: The video information includes: text information and voice information; the obtaining of the semantic time period based on the video information includes: Calculating an edit distance between any two adjacent video shot segments from among the plurality of video shot segments based on text information of the two video shot segments; In response to the edit distance being less than a distance threshold, merging the time periods of the two video shot segments to obtain an intermediate time period; Detecting whether the intermediate time period is within the voice time periods corresponding to the two video lens segments, or overlaps with the voice time periods corresponding to the two video lens segments; In response to detecting that the intermediate time period is within the speech time period corresponding to the two video shot segments or that the intermediate time period overlaps with the speech time period corresponding to the two video shot segments, the intermediate time period is expanded based on the speech time period to obtain a semantic time period.

4. The method according to claim 3, wherein: The obtaining of the semantic time period based on the video information further comprises: In response to detecting that the speech time periods corresponding to the two video shot segments are within the middle time period, the middle time period is used as the semantic time period.

5. The method according to claim 1, wherein: The obtaining of semantic shot segments based on the multiple semantic time periods and the multiple video shot segments comprises: Detecting whether the semantic time period spans across multiple video footage segments; In response to detecting that the semantic time period spans across multiple video shot segments, the multiple video shot segments spanned by the semantic time period are merged to obtain semantic shot segments of the semantic time period.

6. The method according to claim 5, wherein: The obtaining of semantic shot segments based on the semantic time period and the plurality of video shot segments further comprises: In response to detecting that the semantic time period does not span across multiple video shot segments, the video shot segments corresponding to the semantic time period are merged to obtain the semantic shot segments of the semantic time period.

7. The method according to claim 1, further comprising: Based on the semantic shot fragments, labeling information of the video to be processed is obtained.

8. The method according to claim 7, further comprising: In response to receiving the display information of the video to be processed, display points corresponding to the annotation information are displayed on a preview timeline of the video to be processed.

9. The method according to claim 7 or 8, wherein: The annotation information includes: annotation time and annotation type. The annotation information of the video to be processed obtained based on the semantic shot segment includes: Inputting the semantic shot segments into a pre-trained multimodal large model to obtain labeling results of each semantic shot segment output by the multimodal large model; In response to the annotation result being that annotation is required, obtaining an annotation type from the annotation result; The initial time of the semantic shot segment is used as the marking time.

10. The method according to claim 9, further comprising: Detect whether there are two semantic shot segments whose difference in annotation time is less than a time threshold; In response to detecting two semantic shot segments having a difference in annotation time less than a time threshold, detecting whether the annotation types of the two semantic shot segments are the same; In response to detecting that the annotation types of the two semantic shot segments are the same, the annotation types of the two semantic shot segments are merged, and the annotation times of the two semantic shot segments are aggregated.

11. A video processing device, comprising: A shot obtaining unit is configured to perform shot segmentation on the obtained video to be processed to obtain a plurality of video shot segments; an extraction unit, configured to extract video information of each video shot segment from the plurality of video shot segments; A time obtaining unit is configured to obtain a semantic time period based on the video information; The semantic obtaining unit is configured to obtain a semantic shot segment based on the semantic time period and the plurality of video shot segments.

12. The device according to claim 11, wherein The shot obtaining unit is configured to: obtain a video to be processed; input the video to be processed into a pre-trained segmentation model to obtain a plurality of video shot segments output by the segmentation model.

13. The device according to claim 11, wherein: The video information includes: text information and voice information; the time obtaining unit is configured to: calculate the editing distance between any two adjacent video shot segments in a plurality of video shot segments based on the text information of the two video shot segments; in response to the editing distance being less than a distance threshold, merge the time periods of the two video shot segments to obtain an intermediate time period; detect whether the intermediate time period is within the voice time period corresponding to the two video shot segments, or overlaps with the voice time period corresponding to the two video shot segments; in response to detecting that the intermediate time period is within the voice time period corresponding to the two video shot segments, or that the intermediate time period overlaps with the voice time period corresponding to the two video shot segments, expand the intermediate time period based on the voice time period to obtain a semantic time period.

14. The device according to claim 13, wherein: The time obtaining unit is further configured to: in response to detecting that the speech time periods corresponding to the two video shot segments are within the middle time period, use the middle time period as a semantic time period.

15. The device according to claim 11, wherein The semantic obtaining unit is configured to: detect whether the semantic time period spans across multiple video shot segments; in response to detecting that the semantic time period spans across multiple video shot segments, merge the multiple video shot segments spanned by the semantic time period to obtain the semantic shot segment of the semantic time period.

16. The device according to claim 15, wherein: The semantic obtaining unit is further configured to: in response to detecting that the semantic time period does not span across multiple video shot segments, merge the video shot segments corresponding to the semantic time period to obtain the semantic shot segments of the semantic time period.

17. The apparatus according to claim 11, further comprising: The annotation obtaining unit is configured to obtain annotation information of the video to be processed based on the semantic shot segment.

18. The apparatus according to claim 12, further comprising: The dotting unit is configured to display the display points corresponding to the annotation information on the preview timeline of the video to be processed in response to receiving the display information of the video to be processed.

19. The device according to claim 17 or 18, wherein: The annotation information includes: annotation time and annotation type, and the annotation obtaining unit is configured to: input the semantic lens fragment into a pre-trained multimodal large model to obtain the annotation results of each semantic lens fragment output by the multimodal large model; in response to the annotation result being required to be labeled, obtain the annotation type from the annotation result; and use the initial time of the semantic lens fragment as the annotation time.

20. The device according to claim 19, wherein The device also includes: an aggregation unit, which is configured to: detect whether there are two semantic shot segments whose difference in annotation time is less than a time threshold; in response to detecting two semantic shot segments whose difference in annotation time is less than the time threshold, detect whether the annotation types of the two semantic shot segments are the same; in response to detecting that the annotation types of the two semantic shot segments are the same, merge the annotation types of the two semantic shot segments, and aggregate the annotation times of the two semantic shot segments.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images