Video processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and provides a video processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target video to be processed, and extracting a plurality of modal features in the target video; according to the extracted modal features, determining highlight fragments in the target video, and generating respective type labels of the highlight fragments; obtaining candidate processing materials matched with the highlight fragment, and screening out a target processing material from the candidate processing materials based on the type label; and processing the highlight fragment by using the target processing material to generate a to-be-published video.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of short video platforms, more and more users are posting video resources on short video platforms. In order to ensure the high quality of video content, users usually enhance the viewing experience of the video by adding packaging elements (such as stickers, text effects, text templates, and sound effects) to the video.

[0003] However, currently, the packaging of videos is done by users searching for corresponding materials in the material library (stickers, text effects, text templates, and sound effects, etc.) based on their personal experience and preferences, adding the retrieved corresponding materials to the corresponding timeline, and then manually exporting the packaged video. However, relying on manual packaging by users not only makes it difficult to ensure the packaging quality of the video, but also takes a lot of time to search for materials, resulting in low video packaging efficiency. Summary of the invention

[0004] In view of this, one or more embodiments of the present disclosure provide a video processing method, device, electronic device and storage medium to solve the problem of difficult to ensure packaging quality and low efficiency caused by manual video packaging.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, comprising: extracting multiple modal features in a target video to be processed; determining highlight segments in the target video based on the extracted modal features, and generating type labels for the highlight segments; obtaining candidate processing materials that match the highlight segments; based on the type labels, screening out target processing materials from the candidate processing materials; and processing the highlight segments using the target processing materials to generate a video to be released.

[0006] The video processing method provided by the disclosed embodiment extracts multiple modal features in the target video to determine the highlight segments in the target video, thereby realizing the automatic extraction of the highlight segments. The target processing material corresponding to the highlight segments is determined in combination with the type label corresponding to the highlight segments, and then the target processing material and the highlight segments are processed to generate the video to be released, thereby realizing the automatic retrieval of the target processing material, which not only reduces the time cost of manual retrieval and improves the efficiency of video packaging, but also can ensure the matching of the target processing material and the highlight segments to improve the quality of video packaging.

[0007] In a second aspect, an embodiment of the present disclosure provides a video processing device, comprising: a feature extraction module, used to extract multiple modal features in a target video to be processed; a highlight segment recognition module, used to determine the highlight segments in the target video based on the extracted modal features, and generate type labels for the respective highlight segments; a material acquisition module, used to acquire candidate processing materials that match the highlight segments; a material screening module, used to screen out target processing materials from the candidate processing materials based on the type labels; and a video processing module, used to process the highlight segments using the target processing materials to generate a video to be published.

[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the video processing method of the first aspect or any corresponding embodiment thereof is implemented.

[0009] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program. When the computer program is executed by a processor, the video processing method of the first aspect or any corresponding embodiment thereof is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a flowchart of a video processing method according to some embodiments of the present disclosure;

[0012] Figure 2 is a flowchart of another video processing method according to some embodiments of the present disclosure;

[0013] Figure 3 is a schematic diagram of a process of video packaging according to some embodiments of the present disclosure;

[0014] Figure 4 is a flowchart of another video processing method according to some embodiments of the present disclosure;

[0015] Figure 5 is a schematic diagram of audio track processing and video frame processing according to some embodiments of the present disclosure;

[0016] Figure 6is a schematic diagram of displaying a video to be released according to some embodiments of the present disclosure;

[0017] Figure 7 is a structural block diagram of a video processing device according to an embodiment of the present disclosure;

[0018] Figure 8 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0020] As for the packaging of video content, when the content quality is high, users can enhance the viewing experience of the video by adding packaging elements (such as stickers, text effects, text templates and sound effects) to the video, thereby gaining greater user stickiness.

[0021] The video content packaging process in the related art is: the user searches for the corresponding material in the material library (stickers, text effects, text templates and sound effects, etc.), adds the corresponding material to the corresponding timeline based on personal experience and preferences, and then manually exports the packaged video. However, this packaging method has the following defects:

[0022] (1) The cost of searching and matching materials is high, the efficiency of video content packaging is low, and it consumes manpower and time costs;

[0023] (2) The professional level of users is required to be high, and the quality of video packaging is difficult to guarantee;

[0024] (3) Usually, video packaging will have a better effect if it is added to the highlights of the corresponding video. However, for a large number of users, they lack the ability to discover the corresponding video highlights and control the frequency.

[0025] Based on this, the technical solution disclosed in the present invention extracts multiple modal features in the target video to determine the highlight clips and the materials corresponding to the highlight clips in the target video, and packages the target processing materials and the highlight clips, thereby realizing automatic discovery and automatic packaging of video highlights, and improving video packaging efficiency and quality.

[0026] According to an embodiment of the present invention, a video processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a video processing method is provided, which can be used in electronic devices such as computers, mobile phones, notebook computers, etc. Figure 1 is a flow chart of a video processing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0028] Step S101, extracting multiple modal features from the target video to be processed.

[0029] The target video is the video that the user inputs into the electronic device and needs to be packaged. The video can be obtained by the user in real time through the electronic device, can be selected by the user from the storage space of the electronic device, or can be imported by the user through an external storage device (such as a USB flash drive, a mobile hard disk, etc.). Of course, it can also be obtained through other methods, which are not specifically limited here.

[0030] Modal features are element features that exist in the target video and are used to constitute the video, such as sound, natural language, visual images, etc. By analyzing the target video to be processed, the audio data and video data that constitute the target video can be determined, and then the corresponding modal features can be extracted from the audio data and video data.

[0031] Step S102, determining the highlighted segments in the target video according to the extracted modal features, and generating type labels for the highlighted segments.

[0032] The target video is composed of multiple clips, and the highlight clips are the highlight clips in the target video that need to be packaged. Combining the extracted multiple modal features, the highlight clips corresponding to each modal feature can be determined. The type label is used to characterize the type of the highlight clip, such as cute, retro, funny, fresh, etc. The type label can be determined based on the vertical category and vertical knowledge graph corresponding to each modal feature.

[0033] Step S103, obtaining candidate processing materials that match the highlighted segment.

[0034] The candidate processing materials are multiple video packaging materials retrieved from the packaging material library. The candidate processing materials can be presented in the form of a material list. Specifically, the candidate processing materials can include stickers, sound effects, text templates, animations, and the like.

[0035] Step S104: Filter out target processing materials from candidate processing materials based on the type label.

[0036] When the material has a corresponding label and the material label is consistent with the type label of the highlighted segment, it can be used as a candidate processing material. Since the candidate processing material and the highlighted segment are in the same graph system, multiple candidate processing materials corresponding to the highlighted segment can be selected from the packaging material library in combination with the vertical knowledge graph.

[0037] When packaging the highlight clips, the style of the material needs to be consistent with the video packaging style, and the target processing material is the material that is selected from the candidate processing materials and meets the video packaging style.

[0038] Step S105, using the target processing material to process the highlighted segment to generate a video to be released.

[0039] The highlighted clip has a corresponding video frame and audio track. The audio track and video track corresponding to the highlighted clip are packaged in combination with the determined target processing material to add the target processing material to the appropriate position to obtain the packaged audio track and video frame. The packaged audio track and video frame are merged to generate a packaged video, which is the video to be released.

[0040] After obtaining the video to be released, the user can preview the video to be released to determine whether it meets the packaging requirements. If it is determined that the video to be released meets the packaging requirements, the video to be released can be exported for release processing.

[0041] The video processing method provided in this embodiment extracts multiple modal features in the target video to determine the highlight segments in the target video, thereby realizing the automatic extraction of the highlight segments. The target processing material corresponding to the highlight segments is determined in combination with the type label corresponding to the highlight segments, and then the target processing material and the highlight segments are processed to generate the video to be released, thereby realizing the automatic retrieval of the target processing material, which not only reduces the time cost of manual retrieval and improves the efficiency of video packaging, but also ensures the matching of the target processing material and the highlight segments to improve the quality of video packaging.

[0042] In this embodiment, a video processing method is provided, which can be used in electronic devices such as computers, mobile phones, notebook computers, etc. Figure 2 is a flow chart of a video processing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0043] Step S201, extracting multiple modal features from the target video to be processed.

[0044] Specifically, the modal feature includes at least one of a visual feature, an acoustic feature, and a natural language feature. Accordingly, the above step S201 may include:

[0045] Step S2011, determining the type of the target video, and identifying visual features matching the type in the video frame of the target video.

[0046] The type is used to characterize the content type of the target video, for example, it may include variety shows, sports, fashion, commodities, etc.

[0047] Visual features are features of visual angles determined by the business type, that is, identifying the corresponding objects of interest from the target video. Specifically, Figure 3 As shown, here, the target video input by the user can be analyzed to determine the video images or subtitles contained in the target video, and then the visual angle features can be detected from the video images or subtitles through target detection technology to determine the visual features corresponding to the type.

[0048] Step S2012: Detecting acoustic features representing a specified sound event from the audio data of the target video.

[0049] The acoustic feature represents a feature from an acoustic perspective, and the acoustic feature can be characterized by a specified sound event, such as applause, laughter, etc. contained in the audio data.

[0050] Specifically, Figure 3 As shown, here the target video input by the user can be analyzed to determine the audio data contained in the target video, and the sound events contained in the audio data can be detected by combining the Acoustic Events Detection (AED) technology to determine the acoustic features corresponding to the sound events.

[0051] Among them, the sound event detection may include Mel-Frequency Cepstral Coefficients (MFCC) and filter banks (Fbanks), etc. There is no specific limitation on the sound event detection here, and those skilled in the art can set the detection method according to actual needs.

[0052] Step S2013: Generate text information representing the audio data of the target video, and identify natural language features matching the business type of the target video from the text information.

[0053] The text information is the transcribed text corresponding to the audio data, and the natural language features are used to represent the key words contained in the audio data, such as product names.

[0054] Specifically, Figure 3As shown, the corresponding audio data is extracted from the target video, the audio data is converted into text information, and the natural language features corresponding to the business type are extracted from the text information based on the vertical knowledge graph. Of course, the vertical knowledge graph and the bidirectional encoder representation from transformer (BERT) model can also be combined here to extract natural language features from audio data.

[0055] Among them, the BRET model is a supervised model trained with text information as input and focus words as output. The business areas covered by the vertical knowledge graph can mainly include variety shows, sports, fashion, commodities, etc. The natural language features of each business field can be extracted through the BERT model.

[0056] Step S202: Determine the highlighted segments in the target video based on the extracted modal features, and generate type labels for the highlighted segments.

[0057] Specifically, the above step S202 may include:

[0058] Step S2021, for any extracted modal feature, identify the time interval corresponding to the modal feature in the target video, and use the video segment within the time interval as the determined highlight segment.

[0059] For any modal feature among visual features, acoustic features, and natural language features, each modal feature has a corresponding playback timestamp in the target video. Combined with the playback timestamp, the time interval in which each modal feature is played in the target video and the video clip corresponding to each time interval can be determined, and the video clip corresponding to each modal feature can be determined as the highlighted clip.

[0060] Here, modal features are used to extract video clips of corresponding time intervals, thereby achieving automatic discovery of highlight clips.

[0061] Step S2022: Generate type labels for the highlighted segments. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0062] Step S2023, determining weight information of each highlighted segment according to the type label of each highlighted segment.

[0063] The highlighted segments corresponding to each modal feature have corresponding type labels, which are determined based on the business type of the highlighted segments. The vertical industry to which the highlighted segments corresponding to each modal feature belong contains multiple type labels, and different type labels have different importance, thereby determining the weight information used to characterize the importance.

[0064] Step S2024: Filter out target highlighted segments from the highlighted segments based on the weight information.

[0065] like Figure 3 As shown, according to the determined weight information of the highlight segments, all the highlight segments are sorted, the top N target highlight segments are screened out, and the remaining highlight segments are eliminated.

[0066] In a specific implementation, an importance scoring model may be pre-trained to score the importance of each highlighted segment through the importance scoring model, and then the highlighted segments are sorted according to the importance score to select N target highlighted segments with high importance scores.

[0067] Step S203, obtaining candidate processing materials that match the highlighted segment. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0068] Step S204: Filter out target processing materials from candidate processing materials based on the type label.

[0069] Specifically, the above step S204 may include:

[0070] Step S2041, for any type tag, count the number of highlighted segments with the type tag, and determine the processing style of the target video based on the counted number.

[0071] The processing style is used to characterize the packaging style of the target video. Specifically, for the type label of any highlighted segment, its type label may include retro, funny, fresh, cute, etc. The type labels of the highlighted segments corresponding to different modal features are counted respectively to determine the number of each type label and its proportion. Combined with the proportion of each type label, the overall packaging style for the target video is determined.

[0072] Step S2042, selecting target processing materials that meet the processing style from the candidate processing materials.

[0073] The type label of the highlighted clip is consistent with the material label. Figure 3 As shown, for the highlighted segments corresponding to each modal feature, candidate processing materials matching them can be retrieved from the material library, and then target processing materials can be determined from the candidate processing materials.

[0074] For example, the packaging style of the target video is funny. Since each type label corresponding to each modal feature can correspond to many different styles of candidate processing materials, after determining the packaging style of the target video, funny target processing materials can be screened out from the candidate processing materials, and other non-funny materials can be removed.

[0075] Step S205, using the target processing material corresponding to the target highlighted segment, the target highlighted segment is processed to generate a video to be released.

[0076] The determined target processing material is placed at the appropriate position of the target highlight segment to package the target highlight segment and generate a packaged video to be released, thereby highlighting the performance focus in the target video, reducing the density of material packaging and improving the viewing experience of the video to be released.

[0077] The video processing method provided in this embodiment realizes automatic recognition of multiple modal features by identifying visual features, acoustic features, and natural language features from the target video, thereby reducing the professional level requirements for users and facilitating accurate determination of highlight segments from the target video. By detecting the number of highlight segments with type labels to determine the processing style of the target video, the target processing material can be determined based on the processing style, thereby achieving accurate matching of highlight segments with target processing materials, so that the target processing material can be added to the highlight segments to obtain better video guarantee effects. The weight information corresponding to the type labels of the respective highlight segments is combined to determine the target highlight segment from the highlight segment, so that the target highlight segment can be packaged to highlight the performance focus of the video packaging, reduce the density of added materials, and enhance the video viewing experience.

[0078] In this embodiment, a video processing method is provided, which can be used in electronic devices such as computers, mobile phones, notebook computers, etc. Figure 4 is a flow chart of a video processing method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0079] Step S301, obtaining multiple modal features in the target video to be processed. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0080] Step S302: Determine the highlight segments in the target video according to the extracted modal features, and generate type labels for the highlight segments. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0081] Step S303, obtaining candidate processing materials that match the highlighted segment. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0082] Step S304: based on the type label, select the target processing material from the candidate processing materials. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0083] Step S305, using the target processing material to process the highlighted segment to generate a video to be released.

[0084] Specifically, the above step S305 may include:

[0085] Step S3051: Identify audio segments without substantial semantics from the audio data of the target video, and remove the audio segments from the audio data.

[0086] Audio clips without substantive semantics are meaningless audio clips contained in audio data, that is, audio clips without substantive semantics are clips that need to be removed, such as repeated audio clips, multiple repetitions of words, and idiotic clips in audio data. Specifically, audio data is extracted from the target video, and the audio data is tested to identify audio clips without substantive semantics. Then, the audio clips without substantive semantics are removed from the audio data to improve the audio listening experience.

[0087] Step S3052: Process the highlighted segments in the audio data after the audio segments are removed using the corresponding target processing material.

[0088] After removing the meaningless audio segments in the audio data, the audio tracks and video frames of the highlighted segments can be packaged. Specifically, while removing the meaningless audio segments in the audio data, the audio data can also be subjected to noise reduction processing, and the audio tracks in the highlighted segments can be subjected to speed change processing and voice change processing using the target processing material to obtain packaged audio track data, such as Figure 5 As shown. For the video frames in the high-frequency clips, the placement area of ​​the target processing material in the video frame is identified and detected to achieve the packaging of the video frame. The packaged audio track and video frame are merged to obtain the packaged video to be released.

[0089] Specifically, the above step S305 may further include:

[0090] Step S3053: if processing is performed on the video frames in the highlighted segment, the material type of the current target processing material is identified.

[0091] The material type indicates the type of the target processing material, such as special effect type, filter type, subtitle type, etc.

[0092] Step S3054: Determine, in the video frame, a candidate layout area suitable for the target processing material according to the material type.

[0093] The candidate layout area indicates the placement area of ​​the target processing material in the video frame. Different types of target processing materials have different corresponding placement areas in the video frame. For example, for arrow-type target processing materials, they can point to the identified object of interest in the video frame; for special effect-type target processing materials, they can be placed near the object of interest; for subtitle bold type target processing materials, they can be placed at the subtitle position.

[0094] Specifically, the candidate layout area of ​​the target processing material can be determined by big data analysis combined with space detection and deep neural network (DNN) model to optimize the layout position of the target processing material in the video frame, such as Figure 5 shown.

[0095] In an optional embodiment, the step of determining a candidate layout area suitable for the target processing material in a video frame may include: determining a non-occluded area in the video frame, and determining a candidate layout area suitable for the target processing material in a video picture corresponding to the non-occluded area.

[0096] Since the target processing materials in the video frame packaging are mostly related to vision, such as stickers, special effects, subtitles, etc., it is necessary to identify the non-occluded area in the video frame to avoid the target processing materials appearing in the occluded area and achieve occlusion optimization of the target processing materials, such as Figure 5 shown.

[0097] Specifically, here, face detection, saliency detection, and optical character recognition (OCR) can be used to optimize the occlusion of the video frame to determine the video screen corresponding to the non-occluded area. Then, according to the material type of the target processing material, the candidate layout area of ​​the target processing material is determined from the video screen corresponding to the non-occluded area.

[0098] Step S3055: Process the video frame using the target processing material at the candidate layout area.

[0099] The target processing material is placed in the candidate layout area, so as to package the video frame using the target processing material to obtain a packaged video frame.

[0100] In an optional implementation, if processing is performed on video frames in the highlighted segment, the method may further include:

[0101] Step a1, identifying the color system information of the video frame, and adjusting the color system of the current target processing material based on the color system information.

[0102] Step a2, using the target processing material after the color system is adjusted to process the video frames in the highlight segment.

[0103] Perform color detection on the video screen corresponding to the video frame to determine the color information of the video frame. In order to ensure that the color system of the target processing material is consistent with that of the video frame, the color system of the target processing material can be adjusted in combination with the color system information of the video frame to obtain the target processing material with adjusted color system. Then, add the target processing material with adjusted color system to the video screen corresponding to the video frame to achieve color system optimization of the video frame packaging, such as Figure 5 shown.

[0104] In this embodiment, the color scheme of the target processing material is adjusted so that the target processing material of the corresponding color scheme is added to the video frame of the highlight segment, thereby achieving color optimization of the picture and further improving the video viewing experience.

[0105] Step S306, displaying the video to be released, wherein the display page of the video to be released includes at least a video content display area and a material display area.

[0106] Among them, at the current playback progress, if the display content in the video content display area has added materials, the added materials are displayed in the material display area; after the added materials are triggered, the material operation controls are displayed in the display page of the video to be released, and the material operation controls are used to adjust the added materials.

[0107] The video content display area is used to display the materials added to the target video, and the material display area is used to display the available materials for the target video, such as Figure 6 shown.

[0108] After the packaging of the target video is completed, the video to be released is displayed on the display page of the video to be released, so that the user can confirm whether the packaging of the video to be released meets the needs. If the user needs to adjust the material added to the video to be released, the corresponding material can be added at the appropriate position in combination with the playback progress during the playback of the video to be released. Accordingly, the video to be released can respond to the user's trigger operation on the material, add the material selected by the user to the corresponding position in the video to be released, and display the material operation control on the display page of the video to be released, so as to adjust the material added to the video to be released.

[0109] Here, during the playback of the video to be released, it is supported to adjust the material added to the video to be released to achieve flexibility in video packaging.

[0110] The video processing method provided in this embodiment can effectively improve the audio listening experience and further improve the quality of video packaging by removing meaningless audio clips from audio data. The candidate layout area of ​​the target processing material in the video frame is determined by the material type of the target processing material, so that the target processing material can be added to a suitable position to ensure that the video packaging has a better effect.

[0111] In the present embodiment, a video processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0112] This embodiment provides a video processing device, such as Figure 7 As shown, including:

[0113] The feature extraction module 401 is used to extract multiple modal features from the target video to be processed.

[0114] The highlight segment identification module 402 is used to determine the highlight segments in the target video according to the extracted modal features and generate type labels for the respective highlight segments.

[0115] The material acquisition module 403 is used to acquire candidate processing materials that match the highlighted segment.

[0116] The material screening module 404 is used to screen target processing materials from candidate processing materials based on type labels.

[0117] The video processing module 405 is used to process the highlight segment using the target processing material to generate a video to be released.

[0118] In some optional embodiments, the modal feature includes at least one of a visual feature, an acoustic feature, and a natural language feature. The feature extraction module 401 may include:

[0119] The visual feature recognition unit is used to determine the type of the target video and recognize visual features matching the type in the video picture of the target video.

[0120] The acoustic feature detection unit is used to detect acoustic features representing a specified sound event from the audio data of the target video.

[0121] The natural language feature recognition unit is used to generate text information represented by the audio data of the target video, and recognize natural language features matching the business type of the target video from the text information.

[0122] In some optional embodiments, the highlight segment identification module 402 may include:

[0123] The highlight segment determination unit is used to identify the time interval corresponding to any extracted modal feature in the target video, and use the video segment within the time interval as the determined highlight segment.

[0124] The weight determination unit is used to determine the weight information of each highlighted segment according to the type label of each highlighted segment.

[0125] The highlight segment screening unit is used to screen out target highlight segments from the highlight segments based on weight information.

[0126] In some optional embodiments, the material screening module 404 may include:

[0127] The processing style determination unit is used to count the number of highlighted segments with a type label for any type label, and determine the processing style of the target video based on the counted number.

[0128] The material determination unit is used to select target processing materials that meet the processing style from candidate processing materials.

[0129] In some optional embodiments, the video processing device may further include:

[0130] The highlight segment screening module is used to determine the weight information of each highlight segment according to the type label of each highlight segment, and screen out the target highlight segment from the highlight segments based on the weight information.

[0131] Correspondingly, the video processing module 405 is used to process the target highlighted segment using the target processing material corresponding to the target highlighted segment to generate a video to be published.

[0132] In some optional embodiments, the video processing module 405 may include:

[0133] The audio processing unit is used to identify audio segments without substantial semantics from the audio data of the target video, and remove the audio segments without substantial semantics from the audio data.

[0134] The video processing unit is used to process the highlight segment in the audio data after the audio segment is removed using the corresponding target processing material.

[0135] In some optional embodiments, the video processing module 405 may include:

[0136] The material type determination unit is used to identify the material type of the current target processing material when processing the video frames in the highlight segment.

[0137] A material layout determination unit, used to determine, in a video frame, a candidate layout area suitable for a target processing material according to the material type;

[0138] The first video frame processing unit is used to process the video frame using the target processing material at the candidate layout area.

[0139] In some optional implementation modes, the material layout determining unit may include:

[0140] The layout area determination subunit is used to determine a non-occluded area in a video frame, and determine a candidate layout area suitable for a target processing material in a video picture corresponding to the non-occluded area.

[0141] In some optional embodiments, the video processing module 405 may further include:

[0142] The color recognition unit is used to identify the color information of the video frame when processing the video frame in the highlight segment, and adjust the color of the current target processing material based on the color information.

[0143] The second video frame processing unit is used to process the video frames in the highlight segment using the target processing material after the color system is adjusted.

[0144] In some optional embodiments, the video processing device may further include:

[0145] The display module is used to display the video to be released. The display page of the video to be released includes at least a video content display area and a material display area. Among them, at the current playback progress, if the display content in the video content display area has the added material, the added material is displayed in the material display area; after the added material is triggered, the material operation control is displayed in the display page of the video to be released, and the material operation control is used to adjust the added material.

[0146] The further functional description of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0147] The video processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and a memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0148] The video processing device provided in this embodiment extracts multiple modal features in the target video to determine the highlight segments in the target video, thereby realizing the automatic extraction of the highlight segments. The target processing material corresponding to the highlight segments is determined in combination with the type label corresponding to the highlight segments, and then the target processing material and the highlight segments are processed to generate the video to be released, thereby realizing the automatic retrieval of the target processing material, which not only reduces the time cost of manual retrieval and improves the efficiency of video packaging, but also can ensure the matching of the target processing material and the highlight segments to improve the quality of video packaging.

[0149] An embodiment of the present invention further provides an electronic device having Figure 7 The video processing device shown.

[0150] See also Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0151] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0152] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0153] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, and other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0155] The ultrasonic imaging device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0156] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0157] The electronic device also includes a communication interface for data communication between the electronic device and other devices or a communication network.

[0158] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0159] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A video processing method, characterized in that: The method comprises: Extracting multiple modal features from the target video to be processed; Determine the highlighted segments in the target video according to the extracted modal features, and generate type labels for the highlighted segments; Acquire candidate processing materials matching the highlighted segment; Based on the type label, selecting a target processing material from the candidate processing materials; The highlighted segment is processed using the target processing material to generate a video to be released.

2. The method according to claim 1, characterized in that The modal features include at least one of visual features, acoustic features, and natural language features; extracting multiple modal features from the target video to be processed includes: Determining the type of the target video, and identifying visual features matching the type in a video frame of the target video; and / or Detecting acoustic features representing a specified sound event from audio data of the target video; and / or Text information represented by the audio data of the target video is generated, and natural language features matching the business type of the target video are identified from the text information.

3. The method according to claim 1 or 2, characterized in that: Determining a highlight segment in the target video according to the extracted modal features includes: For any extracted modal feature, the time interval corresponding to the modal feature in the target video is identified, and the video segment within the time interval is used as the determined highlight segment.

4. The method according to claim 1, characterized in that: Based on the type label, selecting a target processing material from the candidate processing materials includes: For any type label, count the number of highlighted segments with the type label, and determine the processing style of the target video based on the counted number; Target processing materials that meet the processing style are screened out from the candidate processing materials.

5. The method according to claim 1, characterized in that After generating the type tags of the respective highlighted segments, the method further includes: Determining weight information of each of the highlighted segments according to the type labels of each of the highlighted segments; Based on the weight information, selecting a target highlighted segment from the highlighted segments; Correspondingly, the target highlighted segment is processed using the target processing material corresponding to the target highlighted segment to generate a video to be released.

6. The method according to claim 1, characterized in that The processing of the highlight segment by using the target processing material includes: Identifying audio segments without substantive semantics from the audio data of the target video, and removing the audio segments without substantive semantics from the audio data; The highlighted segments in the audio data after the audio segments are removed are processed using corresponding target processing materials.

7. The method according to claim 1, characterized in that The processing of the highlight segment by using the target processing material includes: If processing is performed on the video frames in the highlighted segment, identifying the material type of the current target processing material; Determining, in the video frame, a candidate layout area suitable for the target processing material according to the material type; The video frame is processed using the target processing material at the candidate layout area.

8. The method according to claim 7, characterized in that Determining a candidate layout area suitable for the target processing material in the video frame includes: A non-occluded area is determined in the video frame, and a candidate layout area adapted to the target processing material is determined in the video picture corresponding to the non-occluded area.

9. The method according to claim 1, characterized in that: When the highlight segment is processed using the target processing material, the method further includes: If processing is performed on the video frame in the highlighted segment, identifying the color information of the video frame; Adjust the color scheme of the current target processing material based on the color scheme information; The video frames in the highlighted segment are processed using the target processing material after the color system is adjusted.

10. The method according to claim 1, characterized in that After generating the video to be published, the method further includes: Displaying the video to be released, wherein the display page of the video to be released includes at least a video content display area and a material display area; Among them, at the current playback progress, if the display content in the video content display area has added materials, the added materials are displayed in the material display area; after the added materials are triggered, material operation controls are displayed in the display page of the video to be released, and the material operation controls are used to adjust the added materials.

11. A video processing device, characterized in that: The device comprises: A feature extraction module, used to extract multiple modal features from the target video to be processed; A highlight segment identification module, used to determine the highlight segments in the target video according to the extracted modal features, and generate type labels for the highlight segments; A material acquisition module, used to acquire candidate processing materials matching the highlighted segment; A material screening module, used for screening target processing materials from the candidate processing materials based on the type label; The video processing module is used to process the highlighted segment using the target processing material to generate a video to be released.

12. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, the video processing method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the video processing method according to any one of claims 1 to 10 is implemented.

Images