Video segmentation method, device, equipment, medium and program product

Abstract

The embodiment of the invention discloses a video segmentation method, device and equipment, a medium and a program product. The method comprises the following steps: acquiring a target video to be segmented, wherein the target video is a video used for introducing a plurality of articles; performing voice recognition on the target video to obtain a target introduction text corresponding to the target video; based on a target network model obtained by pre-training, performing text segmentation and article matching on the target introduction text, and determining a plurality of target text slices containing the effective introduction content and target introduction articles corresponding to the target text slices; and based on the slice position corresponding to each target text slice and the target introduction article, segmenting the target video, and determining a plurality of target video slices containing the effective introduction content and the target introduction article corresponding to each target video slice. Through the technical scheme of the embodiment of the invention, automatic segmentation of the video can be realized, the efficiency of video segmentation is improved, and the accuracy of video segmentation is ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and in particular to a video segmentation method, apparatus, device, medium and program product. Background Art

[0002] With the rapid development of computer technology, live broadcast content has gradually evolved into various forms, among which the anchor can sell items by introducing them in the live broadcast room. The entire live broadcast can be recorded to obtain a video for introducing multiple items.

[0003] At present, it is usually necessary to segment the video to obtain the video slices corresponding to each introduced item, so as to distribute and recommend the video slices to users. However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] Existing video segmentation methods are often manual segmentation, which is time-consuming and labor-intensive, reduces the efficiency of video segmentation, and cannot guarantee the accuracy of video segmentation. Summary of the invention

[0005] The embodiments of the present invention provide a video segmentation method, apparatus, device, medium and program product to realize automatic video segmentation, improve the efficiency of video segmentation, and ensure the accuracy of video segmentation.

[0006] In a first aspect, an embodiment of the present invention provides a video segmentation method, comprising:

[0007] Obtain a target video to be segmented, where the target video is a video for introducing multiple items;

[0008] Performing speech recognition on the target video to obtain a target introduction text corresponding to the target video;

[0009] Based on the target network model obtained by pre-training, the target introduction text is segmented and matched with items to determine a plurality of target text slices containing effective introduction content and a target introduction item corresponding to each target text slice;

[0010] Based on the slice position corresponding to each of the target text slices and the target introduction object, the target video is segmented to determine a plurality of target video slices containing effective introduction content and a target introduction object corresponding to each of the target video slices.

[0011] In a second aspect, an embodiment of the present invention further provides a video segmentation device, including:

[0012] A target video acquisition module, used to acquire a target video to be segmented, wherein the target video is a video for introducing a plurality of items;

[0013] A target introduction text determination module is used to perform speech recognition on the target video to obtain a target introduction text corresponding to the target video;

[0014] A text segmentation and matching module is used to perform text segmentation and item matching on the target introduction text based on the target network model obtained by pre-training, and determine multiple target text slices containing effective introduction content and target introduction items corresponding to each target text slice;

[0015] The video segmentation and matching module is used to segment the target video based on the text segment position corresponding to each target text segment and the target introduction object, and determine multiple target video segments containing effective introduction content and the target introduction object corresponding to each target video segment.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the video segmentation method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a video segmentation method as provided in any embodiment of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the video segmentation method provided by any embodiment of the present invention.

[0022] One embodiment of the above invention has the following advantages or beneficial effects:

[0023] By performing speech recognition on the target video to be segmented, the target introduction text corresponding to the target video is obtained, and based on the target network model obtained by pre-training, the target introduction text is segmented and matched with items, and multiple target text slices containing valid introduction content and the target introduction items corresponding to each target text slice are determined. Based on the slice position corresponding to each target text slice and the target introduction item, the target video is segmented, and multiple target video slices containing valid introduction content and the target introduction items corresponding to each target video slice are determined. By using the target network model obtained by pre-training, the introduction text of the video can be segmented accurately and efficiently, and it is ensured that the segmented target text slices all contain valid introduction content of the item, thereby avoiding the segmentation of invalid text slices that do not contain the item introduction content, thereby improving the accuracy of text segmentation. By using the method of segmenting video text, the automatic segmentation of the video can be indirectly realized, thereby improving the efficiency of video segmentation and ensuring the accuracy of video segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a flow chart of a video segmentation method provided by an embodiment of the present invention;

[0026] Figure 2 is a flow chart of another video segmentation method provided by an embodiment of the present invention;

[0027] Figure 3 is a structural example diagram of a target network model involved in an embodiment of the present invention;

[0028] Figure 4 It is a structural schematic diagram of a video slicing device provided by an embodiment of the present invention;

[0029] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0031] Figure 1 This is a flow chart of a video segmentation method provided by an embodiment of the present invention. This embodiment is applicable to the case where a video is segmented with the introduction of an item as the center, and is particularly applicable to the segmentation of a live video in a live selling scene. The method can be executed by a video segmentation device, which can be implemented by software and / or hardware and integrated into an electronic device. Figure 1 As shown, the method specifically comprises the following steps:

[0032] S110: Obtain a target video to be segmented, where the target video is a video for introducing multiple items.

[0033] The target video refers to a video that needs to be segmented with the object as the center of the video. The target video can be a video obtained by recording the introduction scene of multiple objects. The introduction scene of multiple objects refers to a scene that introduces the objects one by one according to the arrangement order of the multiple objects. For example, the target video can be but is not limited to a live video for introducing objects. The target video includes the introduction content of each object, so it is necessary to segment the target video with the object as the center to obtain target video slices that introduce each object separately.

[0034] S120: Perform voice recognition on the target video to obtain a target introduction text corresponding to the target video.

[0035] The target introduction text refers to the text converted from the introduction voice in the target video. Specifically, the audio of the target video can be converted into the target introduction text in text format using the existing Automatic Speech Recognition (ASR) method.

[0036] S130. Based on the target network model obtained through pre-training, the target introduction text is segmented and matched with items to determine a plurality of target text slices containing effective introduction content and a target introduction item corresponding to each target text slice.

[0037] Among them, the target network model can be a deep neural network model for realizing text segmentation and item matching. Text segmentation refers to the operation of segmenting the target introduction text with the item as the center. Item matching refers to matching the segmented target text slice containing valid introduction content with all the introduced items to determine the target introduction item described by the target text slice. Valid introduction content can refer to introduction content related to the item, that is, explanation content used to introduce the item. Invalid introduction content can refer to content unrelated to the item, such as opening remarks, and interactive content between the anchor and the audience. Target text slice refers to a text slice containing valid introduction content. Target text slice refers to a time segment of text in the target introduction text, that is, part of the text in the target introduction text. The target introduction item can refer to the item introduced by the target text slice. Each target text slice is a text used to introduce an item. Since the target introduction text is a text introducing multiple items, the number of target text slices is multiple. The target text slice corresponds one to one with the target introduction item.

[0038] Specifically, by pre-training the model based on sample data, a target network model that can accurately perform text segmentation and item matching can be obtained, thereby ensuring the accuracy of the target network model processing. In actual application, the target introduction text is input into the pre-trained target network model for text segmentation and item matching, and multiple target text slices containing effective introduction content and target introduction items corresponding to each target text slice are obtained. By using the target network model, video text segmentation and item matching can be performed accurately and quickly, thereby improving the efficiency and accuracy of text segmentation.

[0039] S140. Based on the slice position corresponding to each target text slice and the target introduction item, the target video is segmented to determine a plurality of target video slices containing effective introduction content and the target introduction item corresponding to each target video slice.

[0040] The target video slice refers to a video slice containing effective introduction content. The target video slice refers to a time segment of the target video, that is, a part of the target video. The target introduction item may refer to the item introduced by the target video slice. Each target video slice is a video for introducing an item. Since the target video is a video introducing multiple items, the number of target video slices is multiple. The target video slices correspond to the target introduction items one by one.

[0041] Specifically, according to the slice position of each target text slice, the target video is segmented so that the segmented target video slice has the same segment position as the target text slice, and the target introduction item corresponding to the target text slice is used as the target introduction item corresponding to the corresponding target video slice, thereby indirectly realizing automatic and fast segmentation of the video by segmenting the video text. After multiple target video slices are segmented, the target video slice corresponding to the item of interest to the user can be sent to the user based on the target video slice corresponding to each target introduction item and the item information of the user's interest, thereby completing the distribution and recommendation of the video slices.

[0042] Exemplarily, step S140 may include: for each target text slice, based on the slice start position and slice end position corresponding to the target text slice, determining the start timestamp and end timestamp corresponding to the target text slice; based on the start timestamp and end timestamp corresponding to the target text slice, segmenting the target video to obtain a target video slice corresponding to the target text slice; and determining the target introduction item corresponding to the target text slice as the target introduction item corresponding to the target video slice.

[0043] Specifically, for each target text slice, the timestamp of the slice start position corresponding to the target text slice is used as the start timestamp of the target text slice, and the timestamp of the slice end position is used as the end timestamp of the target text slice. Since the target video and the target introduction text are synchronously corresponding, based on the timestamp of each video frame in the target video, video frame segmentation is performed at the start timestamp corresponding to the target text slice in the target video, and video frame segmentation is performed at the end timestamp, thereby segmenting a section of the target video and obtaining the target video slice corresponding to the target text slice. By only performing video text segmentation, there is no need to consider the video image, thereby improving the efficiency of video segmentation while ensuring the accuracy of video segmentation.

[0044] The technical solution of this embodiment is to perform speech recognition on the target video to be segmented, obtain the target introduction text corresponding to the target video, and perform text segmentation and item matching on the target introduction text based on the target network model obtained by pre-training, determine multiple target text slices containing valid introduction content and the target introduction items corresponding to each target text slice, segment the target video based on the slice position corresponding to each target text slice and the target introduction items, determine multiple target video slices containing valid introduction content and the target introduction items corresponding to each target video slice. By using the target network model obtained by pre-training, the introduction text of the video can be segmented accurately and efficiently, and it is ensured that the segmented target text slices all contain valid introduction content of the item, thereby avoiding the segmentation of invalid text slices that do not contain the item introduction content, thereby improving the accuracy of text segmentation. By using the method of segmenting video text, the automatic segmentation of the video can be indirectly realized, thereby improving the efficiency of video segmentation and ensuring the accuracy of video segmentation.

[0045] Figure 2 A flowchart of another video segmentation method provided by an embodiment of the present invention. In this embodiment, based on the above embodiments, the target network model includes: a text segmentation sub-model, a content detection sub-model and an item matching sub-model, and on this basis, the specific segmentation process of the target introduction text is described in detail. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0046] See also Figure 2 Another video segmentation method provided in this embodiment specifically includes the following steps:

[0047] S210: Obtain a target video to be segmented, where the target video is a video for introducing multiple items.

[0048] S220: Perform voice recognition on the target video to obtain a target introduction text corresponding to the target video.

[0049] S230, based on the text segmentation sub-model in the target network model, segment the target introduction text, determine a plurality of candidate text slices after segmentation and slice feature information of each candidate text slice.

[0050] The text slice to be selected refers to each text slice obtained by segmentation. The text slice to be selected may contain valid introduction content or may not contain valid introduction content. The text slice to be selected refers to a complete paragraph in the target introduction text. A complete paragraph can be composed of one sentence or multiple sentences. Slice feature information refers to the feature information of the entire text slice to be selected, which can be represented by a feature vector. The text segmentation submodel can be a neural network model for segmenting text.

[0051] Specifically, see Figure 3 , input the target introduction text into the text segmentation sub-model in the target network model for segmentation, obtain each candidate text slice output by the text segmentation sub-model and determine the slice feature information of each candidate text slice.

[0052] For example, see Figure 3 , the text slice sub-model may include: a feature extraction module and a boundary prediction module. Among them, the feature extraction module may be a deep learning network for extracting text feature information. For example, the feature extraction module may be but is not limited to Longformer. Longformer is a deep learning model for processing very long text sequences. Longformer improves the self-attention mechanism of the traditional Transformer model, enabling it to process long texts more efficiently. The boundary prediction module may be a deep learning network for predicting the boundaries of text slices (such as the starting position or the ending position of a text slice). For example, the boundary prediction module may be but is not limited to a Transformer model.

[0053] Accordingly, step S230 may include the following steps S231-S234:

[0054] S231. Divide the target introduction text into sentences to obtain an introduction sentence sequence corresponding to the target introduction text.

[0055] Specifically, based on the periods in the target introduction text, the target introduction text is divided into a plurality of different introduction sentences and combined in the original order to obtain an introduction sentence sequence.

[0056] S232. Based on the feature extraction module in the text segmentation sub-model, feature extraction is performed on each introduction sentence in the introduction sentence sequence to obtain sentence feature information of each introduction sentence.

[0057] Specifically, each introductory sentence in the introductory sentence sequence is input into the feature extraction module for feature extraction, and the sentence feature information of each introductory sentence output by the feature extraction module is obtained. For example, each introductory sentence is input into the feature extraction module Longformer in the form of a word sequence, and the beginning of the sequence is represented by the cls tag in the word sequence. The cls tag can be used as an aggregation vector, that is, the output vector of the cls tag refers to the comprehensive representation of the entire input sequence. At this time, the output vector corresponding to the cls tag output by the Longformer is used as the sentence feature information of the introductory sentence.

[0058] S233. Based on the boundary prediction module in the text slice sub-model and the feature information of each introduction sentence, predict whether each introduction sentence is a boundary sentence. The boundary sentence is used to represent the start position or end position of the text slice.

[0059] Specifically, the feature information of all the introduction sentences can be input into the boundary prediction module to predict the start sentence of the text slice, and the introduction sentence belonging to the start of the slice can be obtained. The feature information of all the introduction sentences can also be input into the boundary prediction module to predict the end sentence of the text slice, and the introduction sentence belonging to the end of the slice can be obtained. By predicting the boundary sentences of the text slice, the text segmentation can be performed at the sentence granularity, which further improves the accuracy of text segmentation.

[0060] Exemplarily, step S233 may include: inputting sentence feature information of all introductory sentences into a boundary prediction module in the text segmentation sub-model for global semantic analysis, and determining whether each introductory sentence is a boundary sentence based on the output of the boundary prediction module.

[0061] Among them, the boundary sentence refers to the sentence whose semantics has changed. Since the introduction content of different items is different, the semantic change will occur when switching the introduced items. The boundary sentence can refer to the beginning sentence or the ending sentence of a paragraph. Specifically, the sentence feature information of all the introduction sentences is input into the boundary prediction module. The boundary prediction module identifies the boundary sentences based on the global semantics between the sentences, and obtains and outputs the probability value of each introduction sentence being a boundary sentence. Based on the probability value of each introduction sentence being a boundary sentence output by the boundary prediction module, the introduction sentence with a probability value greater than the preset probability value is determined as a boundary sentence, so that the target introduction sentence belonging to the boundary sentence can be more accurately predicted by analyzing the semantic change, and then the boundary position of the text slice can be obtained.

[0062] S234, aggregate the introduction sentences based on the boundary prediction results of each introduction sentence, and obtain multiple aggregated candidate text slices and segmentation feature information of each candidate text slice.

[0063] Specifically, the boundary sentences in all the introduction sentences are used as the boundaries of the text slices, and the introduction sentences are divided and aggregated to obtain multiple aggregated candidate text slices and segmentation feature information of each candidate text slice. Since the target introduction text is a text used to introduce multiple items, there are multiple candidate text slices. Each candidate text slice may be a valid text slice that introduces an item, or it may be an invalid text slice that does not introduce an item.

[0064] Exemplarily, step S234 may include: determining the target introductory sentences belonging to the boundary sentences based on the boundary prediction results of each introductory sentence; aggregating all the introductory sentences between every two adjacent target introductory sentences in the introductory sentence sequence to obtain multiple aggregated text slices to be selected; and determining the segmentation feature information of each text slice to be selected based on the sentence feature information of each introductory sentence in each text slice to be selected.

[0065] Specifically, based on the boundary prediction module, the target introduction sentences belonging to the boundary sentences can be obtained, and in the introduction sentence sequence, all the introduction sentences between each two adjacent target introduction sentences are aggregated to obtain each aggregated candidate text slice. In other words, the target introduction text is segmented with the target introduction sentence as the slice boundary to obtain multiple candidate text slices. For each candidate text slice, the sentence feature information corresponding to all the introduction sentences contained in the candidate text slice is spliced ​​to obtain the segmentation feature information of the candidate text slice.

[0066] S240. Based on the content detection sub-model in the target network model and the segmentation feature information of each text slice to be selected, detect whether each text slice to be selected contains valid introduction content, and obtain the target text slice containing the valid introduction content.

[0067] The content detection sub-model may be a neural network model for detecting whether the selected text slice contains the content of the item introduction. The architecture of the content detection sub-model may be any binary classification network model.

[0068] Specifically, see Figure 3 , the segmentation feature information of each candidate text slice is input into the content detection sub-model in the target network model. The content detection sub-model determines the probability value of each candidate text slice containing effective introduction content based on the segmentation feature information of each candidate text slice and outputs it. Based on the probability value of each candidate text slice output by the content detection sub-model, the candidate text slice with a probability value greater than a preset probability value is determined as the target text slice containing effective introduction content.

[0069] S250, performing item matching based on the item matching sub-model in the target network model, the segmentation feature information of each target text slice, and the multiple item introduction information corresponding to the target introduction text, to determine the target introduction item corresponding to each target text slice.

[0070] The item matching sub-model may be a deep learning network model for matching text slices with introduced items. For example, the architecture of the item matching sub-model may be any recall network model. The multiple introduced item information corresponding to the target introduction text refers to the item information of the multiple items introduced by the target video.

[0071] Specifically, see Figure 3 For each target text slice, the segmentation feature information of the target text slice and multiple introduction item information are input into the item matching sub-model in the target network model for item matching, and the target introduction item described by the target text slice is determined. For example, the similarity between the target text slice and each introduction item is determined based on the segmentation feature information of the target text slice and each introduction item information, and the introduction item with the highest similarity is determined as the target introduction item described by the target text slice.

[0072] It should be noted that since text slices other than the target text slice are not valid text slices for introducing item information, there is no need to perform item matching on invalid text slices that do not contain valid introduction content, thereby improving the efficiency of item matching, avoiding the interference of invalid text slices, and improving the accuracy of text segmentation and item matching.

[0073] S260. Based on the slice position corresponding to each target text slice and the target introduction item, the target video is segmented to determine multiple target video slices containing effective introduction content and the target introduction item corresponding to each target video slice.

[0074] The technical solution of this embodiment can automatically segment the target introduction text by using the text segmentation sub-model to obtain multiple text slices to be selected after segmentation, and can automatically detect whether each text slice to be selected contains valid introduction content by using the content detection sub-model to obtain the target text slice containing the valid introduction content, and can automatically match the target text slice with the introduction object by using the object matching sub-model to obtain the target introduction object described by each target text slice, thereby improving the efficiency and accuracy of text segmentation, and further improving the efficiency and accuracy of video segmentation.

[0075] On the basis of the above-mentioned technical solutions, the target network model is pre-trained based on sample data, wherein the sample data includes: sample introduction text, multiple introduction item information corresponding to the sample introduction text, multiple actual text slices to be selected corresponding to the sample introduction text, and multiple target actual text slices containing valid introduction content and actual introduction items corresponding to each target actual text slice.

[0076] Among them, the sample introduction text is the text obtained by performing voice recognition on the sample video. The multiple introduction item information corresponding to the sample introduction text refers to the item information of the multiple items introduced in the sample video. The actual text slice to be selected can be a real text slice obtained by manually performing semantic segmentation of the sample introduction text with the item as the center. For example, the actual text slice to be selected is obtained by dividing and aggregating in a manner of annotating boundary sentences. The target actual text slice refers to the actual text slice to be selected that contains effective introduction content. The actual introduction item corresponding to the target actual text slice refers to the introduction item truly described in the target actual text slice, which can be annotated manually.

[0077] Specifically, when the target network model includes: a text segmentation sub-model, a content detection sub-model and an item matching sub-model, the actual text slices to be selected, the target actual text slices and the actual introduction items corresponding to each target actual text slice are used as training labels, and the text segmentation sub-model, the content detection sub-model and the item matching sub-model are trained as a whole to obtain the trained target network model, thereby ensuring the accuracy of text segmentation and item matching of the target network model.

[0078] Exemplarily, the training process of the target network model may include the following steps S201-S203:

[0079] Step S201, mask the target introduction text, and based on the target network model to be trained, perform mask prediction, text segmentation and item matching on the masked target introduction text, determine the mask prediction result, each selected sample text slice after segmentation and the target sample text slice containing valid introduction content and the sample introduction item corresponding to each target sample text slice.

[0080] Specifically, in order to improve the robustness of the model, data enhancement can be performed by randomly masking the input target introduction text. When the target network model to be trained includes: a text segmentation sub-model, a content detection sub-model, and an item matching sub-model, based on the text segmentation sub-model to be trained, mask prediction and segmentation processing are performed on the masked target introduction text to determine the mask prediction result, each selected sample text slice after segmentation, and the slice feature information of each selected sample text slice; based on the content detection sub-model to be trained and the segmentation feature information of each selected sample text slice, detect whether each selected sample text slice contains valid introduction content, and obtain the target sample text slice containing valid introduction content; based on the item matching sub-model to be trained, the segmentation feature information of each target sample text slice and the multiple introduction item information corresponding to the sample introduction text, item matching is performed to determine the sample introduction item corresponding to each target sample text slice.

[0081] Step S202: determine the mask prediction error based on the mask prediction result and the actual mask character, determine the text segmentation error based on the selected sample text slice and the selected actual text slice, determine the content detection error based on the target sample text slice and the target actual text slice, and determine the item matching error based on the sample introduction item and the actual introduction item.

[0082] Specifically, based on the mask loss function, the mask prediction result and the actual mask character, the mask prediction error between the mask prediction result and the actual mask character is determined. Based on the segmentation loss function (such as the cross entropy loss function, etc.), the sample text slice to be selected and the actual text slice to be selected, the text segmentation error between the sample text slice to be selected and the actual text slice to be selected is determined. Based on the content detection loss function (such as the cross entropy loss function, etc.), the target sample text slice and the target actual text slice, the content detection error between the target sample text slice and the target actual text slice is determined. Based on the item matching loss function (such as the NCE (Noise Contrastive Estimation) loss function), the sample introduction item and the actual introduction item, the item matching error between the sample introduction item and the actual introduction item is determined.

[0083] Step S203: Based on the mask prediction error, text segmentation error, content detection error and item matching error, determine the target error corresponding to the sample introduction text, and based on the target error, adjust the network parameters in the target network model to be trained until the training ends when the preset convergence conditions are met.

[0084] Specifically, the mask prediction error, text segmentation error, content detection error and item matching error are weighted and summed, and the summed result is determined as the target error corresponding to the sample introduction text. The target error is back-propagated to the target network model to be trained, that is, the text segmentation sub-model, content detection sub-model and item matching sub-model to be trained, and the network parameters in the text segmentation sub-model, content detection sub-model and item matching sub-model to be trained are adjusted until the preset convergence conditions are met, such as the number of iterations is equal to the preset number, or the training ends when the target error changes tend to be stable, and the target network model after the training is obtained. Since there may be some wrong words, missing words, and sentence segmentation errors in the actual speech recognition process, data enhancement is performed through mask prediction during the model training process, which improves the robustness of the target network model and further improves the accuracy of text segmentation.

[0085] The following is an embodiment of a video segmentation device provided in an embodiment of the present invention. The device and the video segmentation methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the video segmentation device, please refer to the embodiment of the above-mentioned video segmentation method.

[0086] Figure 4 This is a schematic diagram of the structure of a video segmentation device provided by an embodiment of the present invention. This embodiment is applicable to the case where a video is segmented with the introduction of an item as the center, and is particularly applicable to the segmentation of a live video in a live selling scene. Figure 4 As shown, the device specifically includes: a target video acquisition module 410, a target introduction text determination module 420, a text segmentation and matching module 430 and a video segmentation and matching module 440.

[0087] Among them, the target video acquisition module 410 is used to obtain the target video to be segmented, and the target video is a video for introducing multiple items; the target introduction text determination module 420 is used to perform voice recognition on the target video to obtain the target introduction text corresponding to the target video; the text segmentation and matching module 430 is used to perform text segmentation and item matching on the target introduction text based on the target network model obtained by pre-training, and determine multiple target text slices containing effective introduction content and the target introduction items corresponding to each target text slice; the video segmentation and matching module 440 is used to segment the target video based on the text slice position corresponding to each target text slice and the target introduction items, and determine multiple target video slices containing effective introduction content and the target introduction items corresponding to each target video slice.

[0088] The technical solution of this embodiment is to perform speech recognition on the target video to be segmented, obtain the target introduction text corresponding to the target video, and perform text segmentation and item matching on the target introduction text based on the target network model obtained by pre-training, determine multiple target text slices containing valid introduction content and the target introduction items corresponding to each target text slice, segment the target video based on the slice position corresponding to each target text slice and the target introduction items, determine multiple target video slices containing valid introduction content and the target introduction items corresponding to each target video slice. By using the target network model obtained by pre-training, the introduction text of the video can be segmented accurately and efficiently, and it is ensured that the segmented target text slices all contain valid introduction content of the item, thereby avoiding the segmentation of invalid text slices that do not contain the item introduction content, thereby improving the accuracy of text segmentation. By using the method of segmenting video text, the automatic segmentation of the video can be indirectly realized, thereby improving the efficiency of video segmentation and ensuring the accuracy of video segmentation.

[0089] Optionally, the target network model includes: a text segmentation sub-model, a content detection sub-model and an item matching sub-model;

[0090] The text segmentation and matching module 430 includes:

[0091] A candidate text slice determination unit is used to segment the target introduction text based on the text segmentation sub-model, and determine a plurality of candidate text slices after segmentation and slice feature information of each candidate text slice;

[0092] A target text slice determination unit, configured to detect whether each of the candidate text slices contains valid introduction content based on the content detection sub-model and the segmentation feature information of each of the candidate text slices, and obtain a target text slice containing the valid introduction content;

[0093] The target introduction item determination unit is used to perform item matching based on the item matching sub-model, the segmentation feature information of each target text slice and the multiple introduction item information corresponding to the target introduction text, and determine the target introduction item corresponding to each target text slice.

[0094] Optionally, the text segmentation sub-model includes: a feature extraction module and a boundary prediction module;

[0095] The unit for determining the text slice to be selected includes:

[0096] A sentence division subunit, used to divide the target introduction text into sentences to obtain an introduction sentence sequence corresponding to the target introduction text;

[0097] A feature extraction subunit, configured to extract features of each introductory sentence in the introductory sentence sequence based on the feature extraction module, and obtain sentence feature information of each introductory sentence;

[0098] A boundary prediction subunit, used to predict whether each introductory sentence is a boundary sentence based on the boundary prediction module and feature information of each introductory sentence, wherein the boundary sentence is used to represent the start position or end position of a text slice;

[0099] The sentence aggregation subunit is used to aggregate the introduction sentences based on the boundary prediction results of each introduction sentence, and obtain multiple aggregated candidate text slices and segmentation feature information of each of the candidate text slices.

[0100] Optionally, the boundary prediction subunit is specifically used to: input the sentence feature information of all introduction sentences into the boundary prediction module for global semantic analysis, and determine whether each introduction sentence is a boundary sentence based on the output of the boundary prediction module; wherein the boundary sentence refers to a sentence whose semantics has changed.

[0101] Optionally, the sentence aggregation subunit is specifically used to: determine the target introduction sentence belonging to the boundary sentence based on the boundary prediction result of each introduction sentence; aggregate all the introduction sentences between every two adjacent target introduction sentences in the introduction sentence sequence to obtain multiple aggregated text slices to be selected; based on the sentence feature information of each introduction sentence in each of the text slices to be selected, determine the segmentation feature information of each of the text slices to be selected.

[0102] Optionally, the video segmentation and matching module 440 is specifically configured to:

[0103] For each of the target text slices, based on the slice start position and slice end position corresponding to the target text slice, determine the start timestamp and end timestamp corresponding to the target text slice; based on the start timestamp and end timestamp corresponding to the target text slice, segment the target video to obtain the target video slice corresponding to the target text slice; and determine the target introduction item corresponding to the target text slice as the target introduction item corresponding to the target video slice.

[0104] Optionally, the target network model is obtained by training based on sample data in advance, wherein the sample data includes: a sample introduction text, a plurality of introduction item information corresponding to the sample introduction text, a plurality of actual text slices to be selected corresponding to the sample introduction text, a plurality of target actual text slices containing effective introduction content, and an actual introduction item corresponding to each target actual text slice;

[0105] The device also includes: a target network model training module, which is used to:

[0106] The target introduction text is masked, and based on the target network model to be trained, mask prediction, text segmentation and item matching are performed on the masked target introduction text to determine the mask prediction result, each selected sample text slice after segmentation and the target sample text slice containing valid introduction content and the sample introduction item corresponding to each target sample text slice;

[0107] Determining a mask prediction error based on the mask prediction result and the actual mask character, determining a text segmentation error based on the to-be-selected sample text slice and the to-be-selected actual text slice, determining a content detection error based on the target sample text slice and the target actual text slice, and determining an item matching error based on the sample introduction item and the actual introduction item;

[0108] Based on the mask prediction error, the text segmentation error, the content detection error and the object matching error, the target error corresponding to the sample introduction text is determined, and based on the target error, the network parameters in the target network model to be trained are adjusted until the training is terminated when the preset convergence conditions are met.

[0109] The video segmentation device provided in the embodiment of the present invention can execute the video segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the video segmentation method.

[0110] It is worth noting that in the embodiment of the above-mentioned video segmentation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0111] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0112] like Figure 5 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0113] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0114] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0115] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory 30 (RAM) and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0116] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0117] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a video segmentation method provided by any embodiment of the present invention, the method comprising:

[0119] Obtain a target video to be segmented, where the target video is a video for introducing multiple items;

[0120] Performing speech recognition on the target video to obtain a target introduction text corresponding to the target video;

[0121] Based on the target network model obtained by pre-training, the target introduction text is segmented and matched with items to determine a plurality of target text slices containing effective introduction content and a target introduction item corresponding to each target text slice;

[0122] Based on the slice position corresponding to each of the target text slices and the target introduction object, the target video is segmented to determine a plurality of target video slices containing effective introduction content and a target introduction object corresponding to each of the target video slices.

[0123] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the video segmentation method provided by any embodiment of the present invention.

[0124] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the video segmentation method provided in any embodiment of the present invention are implemented. The method includes:

[0125] Obtain a target video to be segmented, where the target video is a video for introducing multiple items;

[0126] Performing speech recognition on the target video to obtain a target introduction text corresponding to the target video;

[0127] Based on the target network model obtained by pre-training, the target introduction text is segmented and matched with items to determine a plurality of target text slices containing effective introduction content and a target introduction item corresponding to each target text slice;

[0128] Based on the slice position corresponding to each of the target text slices and the target introduction object, the target video is segmented to determine a plurality of target video slices containing effective introduction content and a target introduction object corresponding to each of the target video slices.

[0129] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0130] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0131] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0132] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the video segmentation method provided by any embodiment of the present invention.

[0134] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0135] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0136] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A video segmentation method, characterized in that: include: Obtain a target video to be segmented, where the target video is a video for introducing multiple items; Performing speech recognition on the target video to obtain a target introduction text corresponding to the target video; Based on the target network model obtained by pre-training, the target introduction text is segmented and matched with items to determine a plurality of target text slices containing effective introduction content and a target introduction item corresponding to each target text slice; Based on the slice position corresponding to each of the target text slices and the target introduction object, the target video is segmented to determine a plurality of target video slices containing effective introduction content and a target introduction object corresponding to each of the target video slices.

2. The method according to claim 1, characterized in that The target network model includes: a text segmentation sub-model, a content detection sub-model and an item matching sub-model; The target network model obtained based on the pre-training performs text segmentation and item matching on the target introduction text, and determines a plurality of target text slices containing effective introduction content and a target introduction item corresponding to each target text slice, including: Based on the text segmentation sub-model, segment the target introduction text, determine a plurality of to-be-selected text slices after segmentation and slice feature information of each of the to-be-selected text slices; Based on the content detection sub-model and the segmentation feature information of each of the to-be-selected text slices, detect whether each of the to-be-selected text slices contains valid introduction content, and obtain a target text slice containing valid introduction content; Item matching is performed based on the item matching sub-model, the segmentation feature information of each target text slice and the multiple introduction item information corresponding to the target introduction text to determine the target introduction item corresponding to each target text slice.

3. The method according to claim 2, characterized in that The text segmentation sub-model includes: a feature extraction module and a boundary prediction module; The target introduction text is segmented based on the text segmentation sub-model, and a plurality of to-be-selected text segments and segmentation feature information of each of the to-be-selected text segments are determined after segmentation, including: Divide the target introduction text into sentences to obtain an introduction sentence sequence corresponding to the target introduction text; Based on the feature extraction module, feature extraction is performed on each introduction sentence in the introduction sentence sequence to obtain sentence feature information of each introduction sentence; Based on the boundary prediction module and the feature information of each introduction sentence, predict whether each introduction sentence is a boundary sentence, wherein the boundary sentence is used to represent the start position or the end position of the text slice; Based on the boundary prediction result of each introductory sentence, the introductory sentences are aggregated to obtain a plurality of aggregated candidate text slices and segmentation feature information of each of the candidate text slices.

4. The method according to claim 3, characterized in that The step of predicting whether each introductory sentence is a boundary sentence based on the boundary prediction module and the feature information of each introductory sentence includes: The sentence feature information of all the introduction sentences is input into the boundary prediction module for global semantic analysis, and based on the output of the boundary prediction module, it is determined whether each introduction sentence is a boundary sentence; wherein the boundary sentence refers to a sentence whose semantics has changed.

5. The method according to claim 3, characterized in that: The step of aggregating the introductory sentences based on the boundary prediction results of each introductory sentence to obtain a plurality of aggregated candidate text slices and segmentation feature information of each candidate text slice includes: Based on the boundary prediction results of each introduction sentence, determine the target introduction sentence belonging to the boundary sentence; Aggregating all the introduction sentences between every two adjacent target introduction sentences in the introduction sentence sequence to obtain a plurality of aggregated text slices to be selected; Based on the sentence feature information of each introductory sentence in each of the text slices to be selected, the segmentation feature information of each of the text slices to be selected is determined.

6. The method according to claim 1, characterized in that The target video is segmented based on the slice position corresponding to each target text slice and the target introduction item, and a plurality of target video slices containing effective introduction content and a target introduction item corresponding to each target video slice are determined, including: For each of the target text slices, based on the slice start position and slice end position corresponding to the target text slice, determine the start timestamp and end timestamp corresponding to the target text slice; Based on the start timestamp and the end timestamp corresponding to the target text slice, the target video is segmented to obtain a target video slice corresponding to the target text slice; The target introduction item corresponding to the target text slice is determined as the target introduction item corresponding to the target video slice.

7. The method according to any one of claims 1 to 6, characterized in that: The target network model is obtained by training based on sample data in advance, wherein the sample data includes: a sample introduction text, multiple introduction item information corresponding to the sample introduction text, multiple to-be-selected actual text slices corresponding to the sample introduction text, multiple target actual text slices containing valid introduction content, and actual introduction items corresponding to each target actual text slice; The training process of the target network model includes: The target introduction text is masked, and based on the target network model to be trained, mask prediction, text segmentation and item matching are performed on the masked target introduction text to determine the mask prediction result, each selected sample text slice after segmentation and the target sample text slice containing valid introduction content and the sample introduction item corresponding to each target sample text slice; Determining a mask prediction error based on the mask prediction result and the actual mask character, determining a text segmentation error based on the to-be-selected sample text slice and the to-be-selected actual text slice, determining a content detection error based on the target sample text slice and the target actual text slice, and determining an item matching error based on the sample introduction item and the actual introduction item; Based on the mask prediction error, the text segmentation error, the content detection error and the object matching error, the target error corresponding to the sample introduction text is determined, and based on the target error, the network parameters in the target network model to be trained are adjusted until the training is terminated when the preset convergence conditions are met.

8. A video segmentation device, characterized in that: include: A target video acquisition module, used to acquire a target video to be segmented, wherein the target video is a video for introducing a plurality of items; A target introduction text determination module is used to perform speech recognition on the target video to obtain a target introduction text corresponding to the target video; A text segmentation and matching module is used to perform text segmentation and item matching on the target introduction text based on the target network model obtained by pre-training, and determine multiple target text slices containing effective introduction content and target introduction items corresponding to each target text slice; The video segmentation and matching module is used to segment the target video based on the text segment position corresponding to each target text segment and the target introduction object, and determine multiple target video segments containing effective introduction content and the target introduction object corresponding to each target video segment.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video segmentation method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video segmentation method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the video segmentation method as described in any one of claims 1 to 7.

Images