Content aggregation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a content aggregation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining sub-data containing each business object from to-be-processed data; dividing the sub-video into sub-video clips based on the time range associated with each sentence of subtitle in the sub-subtitle set, and extracting each voice frame feature for each sub-video clip; on the basis of the matching result of the business object features, combining the clustering result of the voice frame features to obtain a voice frame set associated with each business object; and obtaining a target voice feature of each business object based on each voice frame set, and aggregating data content associated with each business object in to-be-processed data according to the business object feature of each business object and the target voice feature. Thus, all video contents associated with the business object can be covered in the data contents obtained by aggregating the business object, and the accuracy of content aggregation is improved.

Description

Background Art

[0002] Under the related technology, in order to meet the personalized viewing needs of related objects when watching videos, usually for each business object appearing in the video, the video clips containing the business object are aggregated separately, so that in response to the viewing needs of the related objects for the specified business object in the video, only the video content containing the specified business object in the screen can be presented.

[0003] Taking the video clips corresponding to a business object in the aggregated video as an example, currently, object recognition is usually performed on the business object in the video to determine the video clips corresponding to the business object in the video, and then the video clips corresponding to the business object are aggregated.

[0004] However, since videos usually contain multiple types of data content, when the existing aggregation method is used to aggregate the business objects to obtain the various video clips, it is only possible to one-sidedly aggregate the video clips of the business object in the screen from the level of image content; therefore, it is impossible to guarantee that the various aggregated video clips cover all the video content associated with the business object, which reduces the accuracy of content aggregation and the subsequent video presentation effect. Summary of the invention

[0005] The embodiments of the present application provide a content aggregation method, device, electronic device and storage medium, which are used to cover all video content associated with a business object in the data content aggregated for the business object, thereby improving the accuracy of content aggregation.

[0006] In a first aspect, a content aggregation method is proposed, comprising:

[0007] Obtaining sub-data containing each business object from the data to be processed; the sub-data includes: a sub-video composed of sub-image content and sub-voice content, and a sub-caption set associated with the sub-video;

[0008] Based on the time range associated with each subtitle in the subtitle set, the subvideo is divided into subvideo segments, and for each subvideo segment, the following operations are performed: based on the corresponding image frames, the features of each business object are extracted, and based on the corresponding voice frames, the features of each voice frame are extracted;

[0009] According to the matching between the business object features corresponding to each of the sub-video clips and the business object features of each pre-stored historical business object, combined with the clustering results of the speech frame features in each of the speech frames corresponding to each of the sub-video clips, the speech frame sets associated with each of the business objects are obtained;

[0010] Based on the speech frame sets associated with each of the business objects, the target speech features corresponding to each of the business objects are obtained, and according to the business object features and target speech features corresponding to each of the business objects, the data contents associated with each of the business objects are aggregated in the data to be processed.

[0011] In a second aspect, a content aggregation device is provided, comprising:

[0012] An acquisition unit, used to acquire sub-data containing each business object from the data to be processed; the sub-data includes: a sub-video composed of sub-image content and sub-voice content, and a sub-caption set associated with the sub-video;

[0013] The extraction unit is used to divide the sub-video into sub-video segments based on the time range associated with each subtitle in the sub-subtitle set, and for each sub-video segment, perform the following operations: extracting each business object feature based on each corresponding image frame, and extracting each voice frame feature based on each corresponding voice frame;

[0014] A clustering unit, configured to obtain a speech frame set associated with each of the business objects according to a matching condition between the business object features corresponding to each of the sub-video clips and the business object features of each pre-stored historical business object, combined with a clustering result of speech frame features in each of the speech frames corresponding to each of the sub-video clips;

[0015] An aggregation unit is used to obtain the target speech features corresponding to each business object based on the speech frame sets associated with each business object, and aggregate the data content associated with each business object in the data to be processed according to the business object features and target speech features corresponding to each business object.

[0016] Optionally, when the speech frame sets associated with the business objects are obtained by combining the clustering results of the speech frame features in the speech frames corresponding to the sub-video clips, the clustering unit is used to:

[0017] Determine, in each of the sub-video segments, a target sub-video segment of a known corresponding business object and other sub-video segments of an unknown corresponding business object;

[0018] Based on the target sub-video segment, a speech frame set associated with a corresponding business object is constructed, and the speech frame features corresponding to the other sub-video segments are clustered to obtain a speech frame set marked with the associated business object.

[0019] Optionally, the constructing of a speech frame set associated with a corresponding service object based on the target sub-video segment, the clustering unit is further configured to:

[0020] For each business object corresponding to the target sub-video segment, perform the following operations:

[0021] Determine each speech frame included in the corresponding target sub-video segment, and speech frame features corresponding to each of the speech frames, and use a preset anomaly detection algorithm to determine an abnormal speech frame among the speech frames by detecting the features of each speech frame;

[0022] Based on deleting the speech frames of the abnormal speech frame, a set of associated speech frames is constructed.

[0023] Optionally, when clustering the features of each speech frame corresponding to the other sub-video segments to obtain a speech frame set marked with an associated business object, the clustering unit is used to:

[0024] For each speech frame feature corresponding to the other sub-video segments, a preset clustering algorithm is used to cluster the speech frame features whose similarity meets the set conditions to obtain a feature set of each speech frame;

[0025] Each speech frame feature set obtained based on clustering triggers a labeling request to obtain business objects labeled respectively for the said speech frame feature sets, and according to the correspondence between the speech frame features in the said speech frame feature sets and the speech frames, each speech frame set labeled with related business objects is obtained.

[0026] Optionally, when determining the target sub-video segment of the known corresponding business object, the clustering unit is used to:

[0027] For each sub-video clip, do the following:

[0028] When there are target business object features that are successfully matched with the business object features associated with the sub-video clips among the pre-stored business object features of the historical business objects, determining the target historical business objects corresponding to the target business object features;

[0029] When it is determined that each target historical service object corresponds to the same historical service object, the sub-video segment is determined to be a target sub-video segment of a known corresponding service object.

[0030] Optionally, when it is determined that each target historical business object corresponds to a different historical business object, the clustering unit is used to perform any one of the following operations:

[0031] Determining the sub-video segment as another sub-video segment of an unknown corresponding business object;

[0032] An object labeling request is triggered based on the sub-video segment, and based on the obtained business object labeling result, a designated business object corresponding to the sub-video segment is determined, and the sub-video segment is determined as a target sub-video segment corresponding to the designated business object.

[0033] Optionally, when determining other sub-video segments of unknown corresponding business objects, the clustering unit is used to:

[0034] For each sub-video clip, do the following:

[0035] When there is no target business object feature successfully matched with each business object feature associated with the sub-video segment in the pre-stored business object features of each historical business object, the sub-video segment is determined to be another sub-video segment of an unknown corresponding business object.

[0036] Optionally, the data to be processed includes: a video consisting of image content and voice content, and a subtitle set associated with the video; when aggregating the data content associated with each of the business objects in the data to be processed based on the business object features and target voice features corresponding to each of the business objects, the aggregation unit is used to:

[0037] Based on the business object features of each business object, object matching is performed in the image content, and for each business object, aggregating to obtain a corresponding first video clip set;

[0038] The speech content of the data to be processed is divided into speech frames, and based on the time range corresponding to each of the speech frames, the video is divided into second video segments, and for each of the speech frames, corresponding speech frame features are extracted respectively, and according to the matching conditions between each of the speech frame features and each of the target speech features, aggregating to obtain a second video segment set corresponding to each of the business objects;

[0039] For each business object, based on the corresponding first video segment set and second video segment set, data content aggregated for the business object is obtained.

[0040] Optionally, when extracting the features of each business object based on each corresponding image frame, the extraction unit is used to:

[0041] Determine each image frame included in the corresponding sub-image content, and use a preset region detection algorithm to detect a business object region containing a business object in each image frame;

[0042] A preset feature extraction algorithm is used to extract features from the business object regions in each image frame to obtain corresponding business object features.

[0043] Optionally, when extracting the features of each speech frame based on the corresponding speech frames, the extraction unit is used to:

[0044] Splitting the sub-voice content segment corresponding to the sub-video segment into individual voice frames, and performing windowing processing on each of the split voice frames to obtain processed voice frames;

[0045] For each processed speech frame, a preset speech feature extraction technology is used to extract the corresponding Mel-frequency cepstral coefficient MFCC feature.

[0046] Optionally, when obtaining the target speech features corresponding to each of the business objects based on the speech frame sets associated with each of the business objects, the aggregation unit is configured to include:

[0047] Based on the speech frame sets associated with the business objects, constructing training samples, wherein a training sample includes: sample speech constructed from speech frames, and annotated sample objects;

[0048] Based on the training samples, the preset object speech recognition model is trained for multiple rounds of iterations to obtain a trained target object speech recognition model, and obtain target speech features constructed for each business object within the target object speech recognition model.

[0049] Optionally, after aggregating the data contents associated with each of the business objects in the data to be processed, the device further includes a response unit, the response unit being configured to:

[0050] In response to a content display request triggered by a specified business object in the data to be processed on an associated playback device, the target data content associated with the specified business object is obtained;

[0051] The target data content is sent to the playback device, so that the playback device plays the video data corresponding to the target data content.

[0052] In a third aspect, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method when executing the computer program.

[0053] In a fourth aspect, a computer-readable storage medium is proposed, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0054] In a fifth aspect, a computer program product is proposed, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0055] The beneficial effects of this application are as follows:

[0056] In an embodiment of the present application, a content aggregation method, device, electronic device and storage medium are proposed. For each business object with content aggregation needs, when determining the aggregation results corresponding to each business object within the content range corresponding to the data to be processed, the business object features and target voice features determined respectively for each business object are used to aggregate the related data content in the data to be processed, so that content aggregation can be achieved starting from both image content and voice content, which increases the aggregation basis that can be considered in the content aggregation process, helps to improve the accuracy of content aggregation, so that the data content associated with each business object obtained by the final aggregation can cover all video content related to each business object in the data to be processed.

[0057] Moreover, in the process of constructing the target speech features corresponding to each sample object, according to the sub-data obtained from the data to be processed, the sub-video segments corresponding to each sentence of the subtitle are split, and then the speech frames corresponding to each sub-video segment are extracted to obtain the speech frame features, and the business object features corresponding to each sub-video segment are obtained by extracting the features of each image frame corresponding to each sub-video segment; then, according to the matching between the extracted business object features and the business object features of the pre-stored historical business objects, combined with the clustering results of the speech frame features corresponding to each sub-video segment, the speech frame sets associated with each business object can be obtained; further, with the help of the speech frame sets associated with each business object, the target speech features corresponding to each business object can be obtained. It can be seen that in the process of generating the target speech features corresponding to each business object, there is no need to obtain the speech data corresponding to each business object separately, but by clustering the speech frames in each sub-video segment, the speech features of each business object in the data to be processed can be obtained, which reduces the difficulty of obtaining the speech data of the business object on the one hand, and on the other hand, can ensure that the obtained target speech features can effectively characterize the speech features of the corresponding business objects in the data to be processed.

[0058] In addition, in the specific content aggregation process, the joint effects of business object characteristics and target voice characteristics are taken into consideration at the same time, so that the aggregated data content contains both content related at the image level and content related at the voice level, which provides a guarantee for the content aggregation effect and improves the integrity and accuracy of content aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of possible application scenarios in the embodiments of the present application;

[0060] Figure 2A This is a schematic diagram of the content aggregation process in an embodiment of the present application;

[0061] Figure 2B Schematic diagram of the process of obtaining sub-video segments based on sub-video division in an embodiment of the present application;

[0062] Figure 2C A schematic diagram of the process of performing service feature matching in an embodiment of the present application;

[0063] Figure 2D A schematic diagram of the process of establishing a speech frame set corresponding to each service object in an embodiment of the present application;

[0064] Figure 2E This is a schematic diagram of the structure of the object speech recognition model in the embodiment of the present application;

[0065] Figure 3 This is a schematic diagram of the process of content aggregation for film and television dramas in an embodiment of the present application;

[0066] Figure 4 This is a schematic diagram of the content aggregation effect in the embodiment of this application;

[0067] Figure 5 This is a schematic diagram of the logical structure of the content aggregation device in an embodiment of the present application;

[0068] Figure 6 A schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the hardware composition structure of another electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0071] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0072] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0073] Data to be processed: refers to the data targeted by content aggregation in the embodiments of the present application. The data to be processed includes the video and the subtitle set corresponding to the video. The video includes image content and voice content.

[0074] Sub-data: is a part of the data to be processed. When the business objects included in the data to be processed are known, in order to reduce the data processing pressure, the data containing each business object is extracted from the data to be processed. In the embodiment of the present application, it can be understood that the corresponding business object features and target voice features are first determined for each business object based on the sub-data with a small data volume, and then the business object features and target voice features extracted for each business object are applied to the content aggregation of the large amount of data to be processed, and the data content associated with each business object is determined in the data to be processed.

[0075] Subtitle set: contains each subtitle corresponding to the video in the data to be processed. In the embodiment of the present application, the subtitle set includes each subtitle, and the start playback time and end playback time corresponding to each subtitle, that is, the subtitle set includes each subtitle and the time range corresponding to each subtitle.

[0076] Speech frame feature: refers to the feature extracted from the speech content in the speech frame. In the embodiment of the present application, the proposed speech frame feature may be an MFCC feature.

[0077] The following is a brief introduction to the design concept of the embodiment of the present application:

[0078] Under the related technology, in order to meet the personalized viewing needs of related objects when watching videos, usually for each business object appearing in the video, the video clips containing the business object are aggregated separately, so that in response to the viewing needs of the related objects for the specified business object in the video, only the video content containing the specified business object in the screen can be presented.

[0079] Taking the video clips corresponding to a business object in the aggregated video as an example, currently, object recognition is usually performed on the business object in the video to determine the video clips corresponding to the business object in the video, and then the video clips corresponding to the business object are aggregated.

[0080] For example, face segments corresponding to the same face are usually extracted from the video first, and then the identified faces are recognized to determine the characters included in the video segments at each time period. Finally, for each relevant character, the video segments that include the character in the image are merged.

[0081] However, since videos usually contain multiple types of data content, when the existing aggregation method is used to aggregate the business objects to obtain the various video clips, it is only possible to one-sidedly aggregate the video clips of the business object in the corresponding video screen from the level of image content; therefore, it is impossible to guarantee that the various aggregated video clips cover all the video content associated with the business object, which reduces the content aggregation efficiency and the subsequent video presentation effect.

[0082] For example, suppose there are two people talking in a video, and there are camera angle switches in the video, that is, for a certain person, the person may not appear in the image, but the corresponding video content is related to the person; when the existing aggregation method is used for processing, it will be impossible to aggregate the video segments in which the person does not appear, so after the video segments in which the person appears are aggregated, the video content corresponding to each segment cannot cover the complete character interaction process.

[0083] In view of this, in an embodiment of the present application, a content aggregation method, device, electronic device and storage medium are proposed. For each business object with content aggregation needs, when determining the aggregation results corresponding to each business object within the content range corresponding to the data to be processed, the business object features and target voice features determined respectively for each business object are used to aggregate related data content in the data to be processed, so that content aggregation can be achieved starting from both image content and voice content, which increases the aggregation basis that can be considered in the content aggregation process, helps to improve the accuracy of content aggregation, so that the data content associated with each business object finally aggregated can cover all video content related to each business object in the data to be processed.

[0084] Moreover, in the process of constructing the target speech features corresponding to each sample object, according to the sub-data obtained from the data to be processed, the sub-video segments corresponding to each sentence of the subtitle are split, and then the speech frames corresponding to each sub-video segment are extracted to obtain the speech frame features, and the business object features corresponding to each sub-video segment are obtained by extracting the features of each image frame corresponding to each sub-video segment; then, according to the matching between the extracted business object features and the business object features of the pre-stored historical business objects, combined with the clustering results of the speech frame features corresponding to each sub-video segment, the speech frame sets associated with each business object can be obtained; further, with the help of the speech frame sets associated with each business object, the target speech features corresponding to each business object can be obtained. It can be seen that in the process of generating the target speech features corresponding to each business object, there is no need to obtain the speech data corresponding to each business object separately, but by clustering the speech frames in each sub-video segment, the speech features of each business object in the data to be processed can be obtained, which reduces the difficulty of obtaining the speech data of the business object on the one hand, and on the other hand, can ensure that the obtained target speech features can effectively characterize the speech features of the corresponding business objects in the data to be processed.

[0085] In addition, in the specific content aggregation process, the joint effects of business object characteristics and target voice characteristics are taken into consideration at the same time, so that the aggregated data content contains both content related at the image level and content related at the voice level, which provides a guarantee for the content aggregation effect and improves the integrity and accuracy of content aggregation.

[0086] See also Figure 1 , which is a schematic diagram of a possible application scenario in an embodiment of the present application, includes a processing device 110 and a client device 120 .

[0087] In the embodiment of the present application, the processing device 110 may process the data to be processed in advance, that is, for each designated business object included in the data to be processed, the processing device 110 may respectively aggregate the associated content determined in the data to be processed and perform content aggregation;

[0088] Furthermore, the processing device 110 can subsequently respond to a viewing indication for content associated with a specified business object in the data to be processed triggered by a related object on the client device 120, obtain data content aggregated in advance for the specified business object, and present video content matching the determined data content on the client device 120.

[0089] In a possible implementation, the processing device 110 can feedback the video content to be presented to the client device 120 in response to a content viewing indication triggered by a related object on an operable page of the client device 120, wherein the operable page can belong to a developed application, web page, or mini-program, and the present application does not impose any specific restrictions on this.

[0090] In the embodiment of the present application, the processing device 110 and the client device 120 may communicate via a wired network or a wireless network.

[0091] The processing device 110 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0092] The client device 120 may be, for example, a mobile phone, a computer, an e-book reader, a smart home appliance, a vehicle-mounted terminal, etc.

[0093] The following describes the relevant content aggregation process in combination with several possible application scenarios:

[0094] Scenario 1: Content aggregation for copyrighted videos.

[0095] In the content aggregation scenario illustrated in scenario 1, the processing device 110 may perform content aggregation processing on the copyrighted videos to be put on the shelves before putting the copyrighted videos on the shelves, wherein the videos to be put on the shelves refer to videos that can be played on a maintained playback platform or application in the future, and the copyrighted videos refer to videos that require the purchase of usage rights or require a large amount of cost to produce, which may specifically refer to TV series, movies, animations, documentaries, etc.

[0096] Assume that the processing device 110 performs content aggregation on the copyrighted video X to be put on the shelves, and determines that there are various business objects required for content aggregation in the copyrighted video X. When the copyrighted video X is a TV series, a movie or a documentary, the business objects are described by the real names of the stars and the names of the roles played by the stars; when the video X to be put on the shelves is an animation, the business objects are described by the names of the animation characters.

[0097] Then, for each determined business object, based on the sub-video i obtained from the copyrighted video X, the business object features and target speech features corresponding to each business object are determined;

[0098] Then, based on the business object features and target voice features corresponding to each business object, the data content corresponding to each business object is aggregated in the copyrighted video X.

[0099] Scenario 2: Aggregate content based on videos posted by relevant objects on social platforms.

[0100] In the content aggregation scenario illustrated in scenario 2, the processing device 110 may perform content aggregation on data generated by the relevant objects and required to be uploaded to the social platform in response to processing requirements of the relevant objects, wherein the data uploaded by the relevant objects include videos and subtitle sets corresponding to the videos.

[0101] Assume that the processing device 110 performs content aggregation on the video Y to be released, and determines that there are various business objects in the video Y to be released that require content aggregation.

[0102] Then, for each determined business object, based on the sub-video j obtained from the to-be-published video Y, the business object features and target voice features corresponding to each business object are determined;

[0103] Then, according to the business object features and target voice features corresponding to each business object, the data content corresponding to each business object is aggregated in the video Y to be released.

[0104] It should be noted that the content aggregation method proposed in the present application can be executed by an electronic device, which may be a processing device 110 or a client device 120. In the following description of the present application, only the processing device 110 executing content aggregation will be used as an example for schematic illustration.

[0105] The content aggregation process is described below from the perspective of the processing device 110 in conjunction with the accompanying drawings:

[0106] See also Figure 2A As shown, it is a schematic diagram of the content aggregation process in the embodiment of the present application. Figure 2A , the content aggregation process is explained in detail:

[0107] Step 201: The processing device obtains sub-data including each business object from the data to be processed.

[0108] In an embodiment of the present application, after the processing device determines the data range for content aggregation, that is, after determining the data to be processed, it determines the business objects that need to be targeted for content aggregation from the various objects corresponding to the data to be processed, wherein the data to be processed includes a video composed of image content and voice content, and a subtitle set associated with the video; the various objects corresponding to the data to be processed refer to the various objects included in the image screen in the corresponding video.

[0109] It should be noted that in the embodiments of the present application, according to actual processing needs, all the objects corresponding to the data to be processed can be used as corresponding business objects; or, based on processing experience, the objects involved in the important content and plot in the data to be processed can be selected as corresponding business objects.

[0110] For example, when the data to be processed is a TV series, each appearance object refers to each entertainer in the TV series. That is to say, for a piece of data to be processed, the corresponding range of each appearance object is known.

[0111] For another example, continuing with the above example, according to actual processing needs, the leading actors, co-lead actors and guest actors in the TV series can be determined as corresponding business objects.

[0112] Afterwards, the processing device can obtain sub-data extracted from the data to be processed in the process of obtaining sub-data containing each business object from the data to be processed, wherein the sub-data includes: a sub-video composed of sub-image content and sub-voice content, and a sub-subtitle set associated with the sub-video; each business object appears and speaks in the sub-video.

[0113] In this way, for the sub-data obtained from the data to be processed, it is equivalent to obtaining a small range of data including each business object from the data to be processed, which can reduce the processing pressure when subsequently generating the target voice features corresponding to each business object.

[0114] Step 202: The processing device divides the sub-video into sub-video segments based on the time range associated with each subtitle in the sub-subtitle set, and performs the following operations for each sub-video segment: extracting features of each business object based on corresponding image frames, and extracting features of each speech frame based on corresponding speech frames.

[0115] After the processing device obtains the sub-data, it obtains the sub-video included in the sub-data and the sub-subtitle set corresponding to the sub-video. In order to better distinguish the voice of each business object in the data to be processed, the sub-video is divided into sub-video segments based on the time range associated with each subtitle in the sub-subtitle set.

[0116] It should be noted that, in the embodiment of the present application, the subtitle set can be stored in a subtitle file, and the file format of the subtitle file can be any one of the formats such as (SubRip Text, srt), (S Station Alpha, ssa), (Advanced SubStation Alpha, ass), etc., and the present application does not impose specific restrictions on this. Taking the subtitle file format of srt as an example, the corresponding subtitle set includes the content of each subtitle and the time range corresponding to each subtitle, wherein the time range corresponding to a subtitle is determined by the start playback time and the end playback time of the subtitle.

[0117] For example, see Figure 2B As shown, it is a schematic diagram of the process of obtaining sub-video segments based on sub-video division in an embodiment of the present application. Figure 2B From the description, it can be seen that after obtaining the sub-data including the sub-data and the sub-subtitle set, according to the time range associated with the first two subtitles in the sub-subtitle set, it can be determined that two sub-video segments can be divided, and the corresponding time ranges are 00:05:00,400 to 00:05:15,300, and 00:05:16,400 to 00:05:25,300, respectively. Among them, the timestamps at both ends of the time range represent the corresponding start playback time and end playback time, and the content format of the timestamp can be understood as hour: minute: second, millisecond format.

[0118] Furthermore, continue to combine Figure 2B To illustrate, for the time range of 00:05:00,400 to 00:05:15,300, the image content and audio content that match the time range are divided in the sub-video to obtain the corresponding sub-video segment 1; similarly, for the time range of 00:05:16,400 to 00:05:25,300, the image content and audio content that match the time range are divided in the sub-video to obtain the corresponding sub-video segment 2.

[0119] When extracting the corresponding business object features for each sub-video segment, the processing device determines the image frames included in the corresponding sub-image content, and uses a preset area detection algorithm to detect the business object area containing the business object in each image frame; then uses a preset feature extraction algorithm to extract features from the business object area in each image frame to obtain the corresponding business object features.

[0120] Taking the extraction of facial feature vectors for each image frame as an example, when the processing device identifies the face area in each image frame, it can use a multi-task cascaded convolutional neural network (MTCNN) algorithm to detect the face area appearing in the image frame; further, it can use an additive angular margin loss (ArcFace) based face recognition algorithm to extract features of the person in each face area.

[0121] It should be noted that in the embodiment of the present application, when extracting facial feature vectors for each image frame, the facial feature vector extraction technology under the relevant technology can be used for processing. The processing method of combining the above-mentioned MTCNN algorithm and ArcFace algorithm is only an exemplary processing method proposed in this application, and this application does not make any specific limitations on this.

[0122] The following describes the respective functions of the MTCNN algorithm and the ArcFace algorithm:

[0123] The MTCNN algorithm is a deep learning algorithm for face detection and facial key point detection. The MTCNN algorithm uses a cascade approach and divides face detection and facial key point detection into three stages: the Proposal Network (P-Net) stage, the Refine Network (R-Net) stage, and the Output Network (O-Net) stage.

[0124] Among them, P-Net is used to generate candidate face frames; R-Net is used to screen candidate face frames and remove unqualified frames; O-Net is used to refine the accuracy of face frames and detect facial key points. When training the region detection model built based on the MTCNN algorithm, the corresponding loss function can be constructed based on the output results of each network, combined with the specific corresponding regression and classification problems, and the parameters can be adjusted with the help of the predicted position and confidence of the face frame.

[0125] The ArcFace algorithm for facial feature vector extraction is a deep learning algorithm for face recognition. The core of the ArcFace algorithm is a new loss function that adds a cosine similarity constraint on the basis of the traditional Softmax classifier, making faces of the same category closer in the feature space and the distance between faces of different categories more obvious, which can improve the accuracy and robustness of face recognition in subsequent applications.

[0126] Moreover, with the help of the trained ArcFace model, the facial feature vector (ie, business object feature) of each facial region can be extracted.

[0127] In this way, when extracting business features for each sub-video segment, the processing device first determines the business object area where the business object is located in the corresponding image frames, so that the area where business object feature extraction is required can be accurately located in the image frame, and then the business object features extracted for each image frame are obtained by extracting features from the business object area in each image frame; therefore, by means of first locating the business object area and then performing feature extraction on the determined business object area, the extraction effect of the business object features can be improved and the omission of the extraction of business object features in the image frame can be avoided.

[0128] At the same time, in the process of extracting speech frame features for each sub-video segment based on the corresponding speech frames, the processing device splits the sub-speech content segment corresponding to the sub-video segment into speech frames, and performs windowing processing on each of the split speech frames to obtain processed speech frames; then, for each of the processed speech frames, a preset speech feature extraction technology is used to extract the corresponding Mel-frequency cepstral coefficient (MFCC) features.

[0129] Specifically, when the processing device extracts speech frame features for sub-video segments, it can obtain a corresponding set of speech frame features for each sub-video segment, wherein the set of speech frame features includes: speech frame features corresponding to each of the split speech frames.

[0130] The following takes the extracted speech frame features as MFCC features as an example to illustrate the process of processing the sub-speech content corresponding to each sub-video clip to obtain the corresponding MFCC features:

[0131] For each sub-video segment divided as above, MFCC features are extracted for each speech frame in the sub-video segment, so that the audio signal corresponding to the sub-video segment can be converted into a set of MFCC coefficients, which can be used to analyze the characteristics of the audio signal. The relevant calculation process is as follows:

[0132] The processing device first pre-processes the sub-voice content corresponding to the sub-video segment. The processing method involved includes dividing the sub-voice content (i.e., the audio signal in the sub-video segment) into frames to obtain corresponding voice frames, wherein the time length corresponding to each voice frame can be 20-30ms; according to actual processing needs, there can be some overlap between adjacent voice frames; then windowing processing is performed on each voice frame, wherein commonly used window functions include Hanning window, Hamming window, etc.; the purpose of windowing is to make the voice signal in the corresponding voice frame gradually change to 0 at both ends to reduce the discontinuity at the frame edge.

[0133] Afterwards, the processing device performs fast Fourier transform (FFT) processing on the speech signal corresponding to each speech frame to convert the time domain signal into a frequency domain signal; the obtained frequency domain signal is then passed through a group of Mel filters to obtain the energy value of each frequency band, wherein the center frequency of the Mel filter group is set according to the Mel scale, and the filters are generally set to 20-40; then, the energy value of each frequency band is taken logarithmically to obtain the logarithmic energy; then, the obtained logarithmic energy sequence is subjected to discrete cosine transform (DCT) processing to obtain a group of frequency coefficients, namely, MFCC coefficients.

[0134] In addition, considering that the DC component and high-frequency part in the MFCC coefficient have little effect on distinguishing the signal characteristics, the DC component can be removed and the high-frequency part in the MFCC coefficient can be truncated. Specifically, the processing device removes the first item (i.e., the DC component) in the MFCC coefficient because it represents the average energy of the signal and does not have the effect of distinguishing the characteristics; and because the energy of the high-frequency part of the MFCC coefficient is small and the effect of distinguishing the signal characteristics is small, the high-frequency truncation is performed on the MFCC coefficient. Then, the processed MFCC coefficients are normalized to have the same scale range, and the MFCC coefficients obtained by the final processing are determined as the obtained MFCC features.

[0135] In this way, the features of each speech frame can be extracted for each sub-video segment. That is, after obtaining the sub-video segment corresponding to a sentence of subtitles, the features of each speech frame corresponding to the sentence of subtitles are extracted, which is equivalent to realizing the feature extraction of the speech signal and obtaining the corresponding features of each speech frame. Moreover, with the help of the extracted MFCC features, the speech signal information included in the speech frame can be extracted, which provides a processing basis for differentially distinguishing the speech of different business objects.

[0136] Step 203: The processing device obtains a set of voice frames associated with each business object based on the matching between the business object features corresponding to each sub-video clip and the pre-stored business object features of each historical business object, combined with the clustering results of the voice frame features in each voice frame corresponding to each sub-video clip.

[0137] In the embodiment of the present application, when the processing device determines the matching between the business object features corresponding to each sub-video clip and the pre-stored business object features of each historical business object, the pre-stored business object features of each historical business object are first obtained.

[0138] The business object features corresponding to each pre-stored historical business object may be obtained by extracting features based on images of the collected historical business objects, wherein the historical business objects are compared with the business objects included in the current data to be processed, and may be business objects included in other previously processed data to be processed, or may be previously collected business objects that may appear in the data to be processed.

[0139] For example, when the data to be processed is a film or television work, a group of celebrities and their corresponding pictures (called reference pictures) can be collected in advance, and the facial feature vectors of the reference pictures can be extracted based on the MTCNN algorithm and the ArcFace algorithm, and the correspondence between the facial feature vectors and the celebrities can be stored.

[0140] It should be noted that in the embodiment of the present application, when the business object feature is a face feature vector, the pre-stored business object features of each historical business object can be understood as being stored in a face library, wherein the face library refers to an established database containing multiple face feature vectors, and when facing a face recognition task, comparison and recognition can be performed based on the face library. In a possible implementation, the face library contains multiple face images with known identity information, and each face image corresponds to a face feature vector.

[0141] Afterwards, the processing device specifically determines the matching situation between the business object features corresponding to each sub-video clip and the pre-stored business object features of each historical business object. By calculating the similarity between the business object features corresponding to each sub-video clip and the business object features of each historical business object, the business object identity corresponding to the business object features in the sub-video clip that meet the requirements of the similarity is determined.

[0142] For example, see Figure 2C As shown, it is a schematic diagram of the process of matching business features in an embodiment of the present application. Figure 2C As shown in the figure, when matching the service object features for the sub-video segment 1 in the film and television work, the face feature vectors are extracted from the n image frames included in the sub-video segment 1 according to the face recognition method, wherein the number of faces in the image frame is the same as the total number of face feature vectors extracted for the image frame;

[0143] Furthermore, continue to combine Figure 2CTo illustrate, the facial feature vector extracted based on each image frame is compared with the facial feature vectors of historical business objects pre-stored in the face library, so that for each facial feature vector, the historical business object most similar to the facial feature vector is determined in the pre-stored face library, and the corresponding feature similarity is obtained, and the matching result corresponding to the facial feature vector is obtained based on the obtained feature similarity; specifically, when it is determined that the corresponding feature similarity exceeds a set threshold, it is determined that the match is successful, so the business object corresponding to the facial feature vector can be determined based on the identity information of the matched historical business object.

[0144] It should be noted that, in the embodiment of the present application, for the matching of each business object feature corresponding to each sub-video clip with the business object features of each pre-stored historical business object, there are two possible matching situations. One is that the business object features of the historical business object can be successfully matched. In this case, it is manifested that the similarity between the business feature vectors exceeds the set threshold, and the business object to be identified in the image frame is a historical business object whose features have been previously extracted and stored in a targeted manner; the other is that the business object features of the historical business object cannot be successfully matched. In this case, it is manifested that the similarity between the business object features does not reach the set threshold, that is, the corresponding image frame does not include the historical business object whose features have been previously extracted and stored in a targeted manner.

[0145] Furthermore, see Figure 2D As shown, it is a schematic diagram of the process of establishing the voice frame set corresponding to each service object in the embodiment of the present application. Figure 2D , based on the matching between the business object features in the sub-video segments and the business object features of the pre-stored historical business objects, combined with the clustering results of the speech frame features in each speech frame corresponding to each sub-video segment, the process of obtaining the speech frame set associated with each business object is described:

[0146] Step 2031: The processing device determines, in each sub-video segment, a target sub-video segment with a known corresponding business object and other sub-video segments with unknown corresponding business objects.

[0147] In an embodiment of the present application, the processing device divides each sub-video clip into target sub-video clips of known corresponding business objects and other sub-video clips of unknown corresponding business objects according to whether the business object features corresponding to each sub-video clip can be successfully matched with the business object features of each pre-stored historical business object and whether the sub-video clip corresponds to a unique historical business object.

[0148] In the process of determining the target sub-video clip of the known corresponding business object, the processing device performs the following operations for each sub-video clip: when there are target business object features that successfully match the business object features associated with the sub-video clip among the pre-stored business object features of the historical business objects, determine the target historical business objects corresponding to each target business object feature; and when it is determined that each target historical business object corresponds to the same historical business object, determine the sub-video clip as a target sub-video clip of the known corresponding business object.

[0149] Specifically, when the processing device determines a sub-video clip as a target sub-video clip, the sub-video clip needs to meet the following conditions at the same time: the business object features extracted from each image frame corresponding to the sub-video clip can find the target business object features that match successfully among the business object features of the pre-stored historical business objects; the sub-video clip corresponds to a unique business object.

[0150] It should be noted that, in the embodiment of the present application, considering that a sub-video clip corresponds to a sentence of subtitles, and a sentence of subtitles usually corresponds to only one speaking object, then, for a sub-video clip, it should uniquely correspond to one business object; therefore, when determining a target sub-video clip of a known corresponding business object, it is necessary to determine the uniqueness of the business object matched by the sub-video clip, wherein the known corresponding business object is describing the object determination situation for the target sub-video clip, in other words, the target sub-video clip refers to a sub-video clip that can be determined to correspond to a specific business object.

[0151] In this way, based on the matching between the business object characteristics in the sub-video clip and the business object characteristics of known historical business objects, as well as the uniqueness of the business object determined by the corresponding sub-video clip, the sub-video clip that can determine the specific corresponding business object in each sub-video clip can be determined as the target sub-video clip. This is equivalent to being able to quickly determine the sub-video clip based on the business object characteristics of the pre-stored known object.

[0152] Specifically, after the processing device determines that each target historical business object matches a sub-video clip, when it is determined that each target historical business object corresponds to a different historical business object, the processing device performs any one of the following operations: determining the sub-video clip as another sub-video clip of an unknown corresponding business object; triggering an object labeling request based on the sub-video clip, and determining the designated business object corresponding to the sub-video clip based on the acquired business object labeling result; and determining the sub-video clip as a target sub-video clip corresponding to the designated business object.

[0153] Specifically, taking the processing device processing the sub-video segment X as an example, after extracting each business object feature from each image frame corresponding to the sub-video segment X, it is determined that each extracted business object feature can find a successfully matched target business object feature in the pre-stored business object features of each historical business object, but when each target business object feature corresponds to a different historical business object, it is impossible to determine the business object that uniquely corresponds to the sub-video segment X;

[0154] In this case, according to actual processing needs, sub-video clip X can be directly determined as other sub-video clips of unknown corresponding business objects, wherein the unknown corresponding business objects are describing the object determination situation of other sub-video clips. In other words, other sub-video clips refer to sub-video clips for which it is not possible to determine which specific business objects they correspond to; or, according to actual processing needs, a request can be made to indicate the specified business object corresponding to sub-video clip X, so that the business object corresponding to sub-video clip X can be uniquely determined, and then sub-video clip X can be determined as a target sub-video clip corresponding to the specified business object.

[0155] In this way, adaptive processing can be performed according to actual processing needs for different process results existing in the determination process of the target sub-video clip; from the perspective of machine processing, sub-video clips corresponding to different historical business objects can be directly determined as other sub-video clips of unknown corresponding business objects, which can ensure the simplicity and efficiency of the processing process, and determining the designated business object corresponding to the sub-video clip by means of manual indication can better utilize the obtained matching results.

[0156] Correspondingly, in the process of the processing device determining other sub-video clips of the unknown corresponding business object, the following operations are performed for each sub-video clip: when there are no target business object features that successfully match the business object features associated with the sub-video clip among the pre-stored business object features of the historical business objects, the sub-video clip is determined to be another sub-video clip of the unknown corresponding business object.

[0157] Specifically, the processing device may directly determine the sub-video segments that meet the following conditions as other sub-video segments of unknown corresponding business objects: among the pre-stored business object features of each historical business object, there is no feature that successfully matches the business object features extracted based on the sub-video segment. In other words, the business object included in the sub-video segment is not within the scope of known business objects.

[0158] In this way, according to the pre-stored business object features of each historical business object, the sub-video segments that cannot match and determine the corresponding business objects in each sub-video segment can be directly determined as other video segments that cannot determine the corresponding business objects.

[0159] Step 2032: The processing device constructs a speech frame set associated with a corresponding business object based on the target sub-video segment, and clusters the speech frame features corresponding to other sub-video segments to obtain a speech frame set marked with associated business objects.

[0160] In the embodiment of the present application, the processing device can select different clustering methods for the target sub-video segment and other sub-video segments to construct a speech frame set corresponding to the business object.

[0161] For target sub-video clips with known corresponding business objects, when constructing a speech frame set associated with the corresponding business object, it is possible to ensure that the image content and speech content corresponding to each sub-video clip correspond to the same business object, that is, for the sub-video clip, the business object in the image content and the business object that speaks the corresponding subtitle are the same object; the processing device can directly cluster the target sub-video clips corresponding to the same business object, and for each known business object, based on the speech frames included in the corresponding target sub-video clip, construct a corresponding speech frame set, wherein the known business object refers to a business object whose corresponding business object feature matches the business object feature of a pre-stored historical business object, and it is determined that there is a business object of the corresponding target sub-video clip; a known business object usually corresponds to multiple target sub-video clips.

[0162] However, considering that in a sub-video clip there may be a situation where the business object corresponding to the image content is different from the business object corresponding to the voice content; based on this, it can be seen that even if the identity of the business object corresponding to the sub-video clip can be determined based on the business object characteristics of the pre-stored historical business objects, there may still be a problem that the identified business object is not actually the speaking object corresponding to the voice content; therefore, after determining the target sub-video clip corresponding to the known business object, if a voice frame set associated with the known business object is constructed directly based on the voice frames included in the corresponding target sub-video clip, there may be a problem of the voice of other business objects being mixed into the voice frame set of the known business object.

[0163] Based on this, when the processing device constructs a speech frame set associated with the corresponding business object based on the target sub-video segment, the following operations can be performed for each business object corresponding to the target sub-video segment: determine the speech frames included in the corresponding target sub-video segment, and the speech frame features corresponding to each speech frame, and use a preset anomaly detection algorithm to determine the abnormal speech frames in each speech frame by detecting the features of each speech frame; and then construct an associated speech frame set based on the speech frames from which the abnormal speech frames have been deleted.

[0164] Specifically, the processing device determines the voice frames included in the corresponding target sub-video for each business object corresponding to the target sub-video segment, deletes abnormal voice frames belonging to other business objects from the corresponding voice frames, and constructs a voice frame set associated with the business object based on the voice frames from which the abnormal voice frames have been deleted.

[0165] When detecting abnormal speech frames, the processing device may use a preset abnormality detection algorithm to identify abnormal speech frames based on speech frame features corresponding to each speech frame.

[0166] The following uses the anomaly detection technology to perform anomaly detection, and the speech frame features are specifically MFCC features as an example to illustrate the process of identifying abnormal speech frames:

[0167] It should be noted that anomaly detection technology can find those observations that are different from other data in a data set, which are so-called outliers or outliers. The present application can specifically use the local outlier factor (LOF) algorithm for anomaly detection. The basic idea of ​​the algorithm is to compare each sample point with the sample points around it, and determine whether the sample point is an outlier by calculating the distance relationship between its local density and the neighborhood.

[0168] Based on the processing idea of ​​the LOF algorithm, combined with the specific processing content, it can be seen that when determining the abnormal speech frames in each speech frame, the features of each speech frame are determined as each sample point. For each sample point, the distance between it and its k nearest neighboring points is calculated, and these distance values ​​are arranged in order from small to large, and the kth distance value is taken as the reachability distance of the point. Then for each sample point, the local density of its neighborhood (i.e., the set of sample points whose distance to the point does not exceed a given radius r) is calculated, that is, the number of sample points in the neighborhood. At the same time, for each sample point in the neighborhood, the maximum value of its reachable distance to the determined sample point is calculated as the local reachability density of the sample point. Afterwards, for each sample point, its LOF value (local outlier factor) is calculated, that is, the average value of the ratio of the local reachability density of the sample point to the local reachability density of its neighborhood, where the larger the LOF value, the more likely the corresponding sample point is to be an abnormal point.

[0169] Furthermore, the abnormal speech frame is determined by calculating the LOF value corresponding to each speech frame feature.

[0170] In this way, considering that there are usually multiple sub-video clips corresponding to each business object in the sub-video, the extraction of speech frame features can be sufficiently guaranteed. Therefore, for each known business object, with the help of anomaly detection algorithm, abnormal speech features different from most speech features can be detected in the speech frame features corresponding to each speech frame, and then the speech frames corresponding to the abnormal speech features are determined as abnormal speech frames, so that the speech frames belonging to other business objects can be effectively identified; avoiding speech frame clustering anomalies caused by the business object recognized from the image of the sub-video clip is different from the business object recognized from the speech, which is equivalent to filtering the corresponding speech frames by detecting the abnormal conditions of each speech frame feature.

[0171] In an embodiment of the present application, in the process of clustering the speech frame features corresponding to other sub-video clips by the processing device to obtain speech frame sets marked with related business objects, a preset clustering algorithm is used for the speech frame features corresponding to other sub-video clips to cluster the speech frame features whose similarity meets the set conditions to obtain the speech frame feature sets; then, a labeling request is triggered based on the speech frame feature sets obtained by clustering to obtain the business objects respectively marked for each speech frame feature set; and based on the correspondence between the speech frame features in each speech frame feature set and the speech frames, each speech frame set marked with the related business objects is obtained.

[0172] Specifically, when the processing device clusters the speech frame features whose similarity meets the set conditions among the speech frame features corresponding to other sub-video clips to obtain the feature sets of each speech frame, it can adopt a density-based clustering algorithm to construct corresponding speech clusters for each speech frame feature corresponding to other sub-video clips, and perform the following operations for each unclustered speech cluster until the clustering processing is completed for all speech clusters: according to the spatial distribution of each speech frame feature corresponding to each speech cluster, search for the direct neighboring cluster that matches the speech cluster, and search for the indirect neighboring cluster that matches the direct neighboring cluster; cluster the found direct neighboring clusters and indirect neighboring clusters into speech clusters, and obtain the corresponding speech frame feature set based on the speech frame features covered by the clustered speech cluster.

[0173] The following takes the density-based spatial clustering of applications with noise (DBSCAN) algorithm as an example to illustrate the process of clustering other sub-video clips to obtain speech frame sets labeled with related business objects:

[0174] For sub-video clips that have corresponding business object features but cannot identify the specific business objects corresponding to the business object features, they can be called other sub-video clips (i.e., sub-video clips that can identify the business objects but cannot determine who the business objects are). Considering that other sub-video clips as a whole may correspond to the voices of multiple business objects, the voice frame features of each voice frame included in the other sub-video clips as a whole can be clustered, and then the specific business object corresponding to each cluster obtained by clustering can be marked.

[0175] It should be noted that DBSCAN is a density-based clustering algorithm that can be used to discover clusters of any shape and identify noise points. The DBSCAN algorithm does not require the number of clusters to be specified in advance, but automatically determines the number and shape of clusters based on the density distribution of the data;

[0176] The basic processing flow corresponding to the DBSCAN algorithm is: construct a data set based on the speech frame features corresponding to each speech frame in other sub-videos. For a given data set, first treat each data point as a cluster, where one data point corresponds to one speech frame feature; then randomly select a data point from the data set as a seed point, find all the data points in its neighborhood, and add them to the cluster. Then, for each data point in the cluster, find all the data points in its neighborhood and add them to the cluster. Repeat this process until all data points in the cluster can no longer find new neighboring points. Then, select a data point from the data set that has not been visited as a new seed point, and repeat the above process until all data points have been visited.

[0177] Finally, after clustering using the DBSCAN algorithm, multiple clusters can be obtained, where each cluster includes the clustered speech frame features, and each cluster corresponds to a business object to be labeled; then, with the help of the various clusters obtained by clustering, the labeling request can be triggered to obtain the labeling results.

[0178] For example, when the sub-video clip corresponds to a film or TV drama clip, each cluster obtained by clustering may correspond to a potential celebrity. By sending each cluster to operations or outsourcing for labeling, the labeled celebrity can be obtained. If it is a new star, you can choose to store the relevant information of the new star in the face library, so that when you perform content aggregation on other data to be processed later, you can perform identity recognition based on the updated face library.

[0179] In this way, for other sub-video clips whose corresponding business objects are unknown, by aggregating and requesting annotation of speech frame features, it is possible to implement annotation of business objects when the business objects corresponding to the sub-video clips are not known historical business objects.

[0180] In summary, during the processing of steps 2031-2032, the processing device can determine the degree of clustering corresponding to the sub-video clips based on the feature matching between the business object features corresponding to the sub-video clips and the business object features of the pre-stored historical business objects. For target sub-video clips of known corresponding business objects, the speech frames corresponding to the target sub-video clips can be directly clustered; and for other sub-video clips of unknown corresponding business objects, the corresponding speech frame features can be clustered as a whole to obtain each speech frame feature set, so that the associated speech frame set can be determined for different business objects in the end.

[0181] In this way, different processing methods can be used for business objects corresponding to different matching results, and the voice frame sets associated with each business object can be determined respectively; when the identity of the business object can be determined, that is, the business object corresponding to the sub-video clip is a historical business object, the voice frame set associated with the business object can be generated by directly clustering the voice frames in the corresponding target sub-video clip and deleting abnormal voice frames; and when the identity of the business object cannot be determined, that is, the business object corresponding to the sub-video clip is not a historical business object, the identity of the business object can also be determined by voice feature clustering and manual participation in annotation, and its associated voice frame set can be determined. Therefore, when facing new and old business objects, voice frame sets can be constructed in a targeted manner, providing a generation basis for generating target voice features corresponding to each business object.

[0182] Step 204: The processing device obtains the target speech features corresponding to each business object based on the speech frame sets associated with each business object, and aggregates the data content associated with each business object in the data to be processed according to the business object features and target speech features corresponding to each business object.

[0183] When executing step 204, in some possible implementations, the corresponding target speech features can be determined directly based on the speech frame sets associated with each business object by calculating the mean of the features of each speech frame in the corresponding speech frame set, and then content aggregation can be achieved based on the target speech features and business object features corresponding to each business object.

[0184] In some other possible embodiments, deep learning can be used to generate training samples based on the speech frame sets associated with each business object, and a speaking object recognition model can be trained based on the obtained training samples to obtain target speech features generated for each business object within the model, and then content aggregation can be achieved with the help of a target speaking object recognition model that has each target speech feature generated internally, as well as business object features corresponding to each business object.

[0185] The following describes two methods that can be used when executing step 204:

[0186] Method 1: directly determine the target speech features corresponding to each business object by using the speech frame set associated with each business object.

[0187] In the implementation process of method one, after the processing device obtains the speech frame set corresponding to each business object, it performs the following operations for each business object: based on the corresponding speech frame set, determines the corresponding speech frame feature set, calculates the feature mean of each speech frame feature in the speech frame feature set, and determines the calculated feature mean as the corresponding target speech feature, wherein the feature mean is obtained by calculating the element mean of each feature position in each speech frame feature, and then combining the element means based on each feature position.

[0188] At the same time, the processing device obtains the business object features corresponding to each business object, which are comprehensively determined when matching features with historical business objects and when constructing a speech frame set associated with the business object. It can then determine the business object features and target speech features corresponding to each business object.

[0189] Based on this, when the data to be processed includes: a video consisting of image content and voice content, and a subtitle set associated with the video, the processing device can perform object matching in the image content based on the business object features of each business object in the process of content aggregation of the data to be processed, and aggregate the corresponding first video segment set for each business object; at the same time, the voice content of the data to be processed is divided into voice frames, and based on the time range corresponding to each voice frame, the video is divided into second video segments, and for each voice frame, the corresponding voice frame features are extracted respectively, and according to the matching situation of each voice frame feature with each target voice feature, the second video segment set corresponding to each business object is aggregated; and for each business object, based on the corresponding first video segment set and second video segment set, the data content aggregated for the business object is obtained.

[0190] Specifically, when the processing device aggregates to obtain the first video clip, it can extract the business object for each video frame in the image content, obtain the business object features, and determine the corresponding business object; then, for each business object, the first video segments including the business object in the picture are cut out from the image content to obtain a first video segment set, wherein in the first video segment set corresponding to a business object, the time lengths corresponding to different first video segments may be different.

[0191] When the processing device aggregates to obtain the second video segment set, it first divides the video corresponding to the data to be processed into second video segments according to the time range corresponding to each subtitle in the subtitle set, and then extracts the features of each speech frame corresponding to each video segment; thereafter, for each business object, in each second video segment, based on the matching of each speech frame feature corresponding to the second video segment with the target speech feature of the business object, the second video segment set corresponding to the business object is determined.

[0192] Afterwards, for each business object, the intersection of the corresponding first video segment set and the second video segment set is determined to obtain an aggregation result of the business object.

[0193] It should be noted that, taking the aggregation result corresponding to a business object as an example, according to actual processing needs, aggregation can be performed on the time period covered by the intersection of the first video clip set and the second video clip set to obtain the time aggregation result corresponding to each business object, or, based on the intersection of the first video clip set and the second video clip set, aggregation can be performed to obtain a video clip set including specific content.

[0194] In this way, without constructing an additional neural network model, the corresponding target speech features within the range of the data to be processed can be directly generated based on the speech frame sets corresponding to each business object extracted from the data to be processed. Then, by starting from the image content and audio content, the content in the data to be processed is aggregated at different levels, which provides a guarantee for the content aggregation effect and improves the integrity and accuracy of the content aggregation.

[0195] Method 2: Determine the target speech features corresponding to each business object with the help of a neural network model.

[0196] In an embodiment of the present application, the processing device can construct each training sample based on the speech frame set associated with each business object in the process of obtaining the target speech features corresponding to each business object based on the speech frame set associated with each business object, wherein a training sample includes: a sample speech constructed by speech frames, and a sample object annotated for the sample speech; based on each training sample, a preset object speech recognition model is iteratively trained for multiple rounds to obtain a trained target object speech recognition model, and to obtain the target speech features constructed for each business object inside the target object speech recognition model.

[0197] Specifically, in order to generate the target speech features corresponding to each business object, the processing device may construct a speech frame set corresponding to each business object, and generate each training sample according to the speech frame set corresponding to each sample object.

[0198] It should be noted that, since the model structure of the initial object speech recognition model constructed in the embodiment of the present application is flexible, training samples can be adaptively constructed according to the actual model structure. In a possible training process, positive samples and negative samples can be generated according to a set ratio according to actual processing needs, wherein the value of the set ratio is set according to actual processing needs, and the present application does not impose any specific restrictions on this; a positive sample refers to a sample constructed according to the correspondence between a speech frame and a business object, usually with the business object as the labeled sample object, and the corresponding sample speech is determined based on the speech frame corresponding to the business object; a negative sample refers to a sample constructed by speech frames and business objects that do not have a corresponding relationship, and a speech frame can be obtained from a speech frame set of a business object as a sample speech frame, and other business objects can be labeled as sample objects corresponding to the sample speech frame.

[0199] In addition, for the object speech recognition model constructed in this application, it can be understood that the speaker recognition (Speaker Recognition, also known as voiceprint recognition) is performed through the model, and the speaker corresponding to the speech segment can be identified according to the voiceprint characteristics of the speech to be recognized. Due to the different shapes, sizes and positions of the vocal organs of each business object, the vocal cord tension and sound frequency are different. It can be understood that the most important task of speaker recognition is to be able to identify which speaker it comes from based on an audio given a range of speakers.

[0200] In the embodiment of the present application, the initial object speech recognition model can be constructed based on any one of the identity vector (Identity Vector, i-vector) algorithm structure, the d-vector algorithm structure, and the x-vector algorithm structure. Among them, the main idea of ​​the i-vector algorithm is to input the MFCC feature sequence of the speech signal into the Gaussian mixture model (GMM), and then calculate the i-vector according to the GMM model parameters. The advantage of the i-vector algorithm is that it can effectively reduce the dimension and improve the accuracy of speech recognition; d-vector is the earliest speaker recognition algorithm based on deep neural network (Deep-Learning Neural Network, DNN). The core idea is to use the real speaker identity of the sample speech as the label of the sample speech during the training stage, which converts the model training into a classification problem. X-vector is an improvement of the d-vector algorithm, which connects the mean and standard deviation of the frame-level features of the audio as segment-level features, and finally classifies the segment-level features into specific speaking objects through a standard feedforward network.

[0201] The following uses x-vector to build an initial object speech recognition model as an example to illustrate the model content processing process:

[0202] When the x-vector architecture is used to construct the initial object speech recognition model, the constructed training samples are sample speech consisting of speech frames belonging to the same sub-video segment and sample objects annotated for the sample speech.

[0203] See also Figure 2E As shown in the figure, it is a schematic diagram of the structure of the object speech recognition model in the embodiment of the present application. Figure 2E , the process of calculating the target speech features is explained:

[0204] When the x-vector algorithm structure is used for processing, MFCC feature extraction is performed on each speech frame included in the input sample speech to obtain corresponding speech frame features (i.e., each MCFF feature), wherein MFCC is a feature commonly used in speech signal processing, which can extract frequency and energy information of audio signals; each MFCC feature is then processed with the help of BiRNN, and then, after being processed by a statistics pooling layer, the mean and variance of the BiRNN output of each speech frame in the sample speech are taken, and then the obtained mean and variance are concatenated to realize sequence-level feature extraction, wherein BiRNN is a neural network capable of processing sequence data, which has hidden layer states in both forward and backward directions, and can take context information into account when processing the sequence; the statistics pooling layer is usually a global average pooling layer, which can average the features of all frames in the input sequence to obtain a vector of fixed length.

[0205] Afterwards, with the help of BiRNN, the corresponding speaker is predicted based on the concatenated mean and variance, and finally Figure 2E Each feature of the x-vector layer can be regarded as the target speech feature corresponding to each speaker. In addition, in order to reduce the amount of calculation and model complexity, the features of the x-vector layer are usually reduced in dimension. Specifically, a fully connected layer can be used to achieve dimensionality reduction, mapping the x-vector to a vector of lower dimension. Then, the features of the x-vector layer after dimensionality reduction are used as the final output.

[0206] During the training process, a softmax classifier is usually used to classify the target speech features of different speakers, predict the corresponding speaking objects, and then adjust the model parameters by calculating the model loss, wherein the loss function adopted can be a cross entropy loss function; then, after the model training results, the x-vector (i.e., target speech features) constructed by the model for each sample object, i.e., each business object can be obtained.

[0207] In this way, by using each speech frame corresponding to each business object to construct a training sample, and training the speaking object recognition model based on the constructed training samples, the speech features of each business object can be more deeply explored with the help of the model that can realize speaking object recognition, thereby improving the reliability of the generated target speech features and ensuring the accuracy of speaking object identification based on the target speech features.

[0208] Afterwards, when the processing device aggregates to obtain the first video segment set, it can extract business objects for each video frame in the image content, obtain business object features, and determine the corresponding business objects. For each business object, it can cut out the first video segments in the picture including the business object from the image content to obtain the first video segment set, wherein in the first video segment set corresponding to a business object, the time lengths corresponding to different first video segments may be different.

[0209] When processing the second set of video clips aggregated by the device, the trained target speech recognition model can be directly used to classify and determine the corresponding specific business objects based on the matching of the speech features extracted from the speech content corresponding to each subtitle and the target speech features constructed internally for each business object.

[0210] Afterwards, for each business object, the intersection of the corresponding first video segment set and the second video segment set is determined to obtain an aggregation result of the business object.

[0211] It should be noted that, taking the aggregation result corresponding to a business object as an example, according to actual processing needs, aggregation can be performed on the time period covered by the intersection of the first video clip set and the second video clip set to obtain the time aggregation result corresponding to each business object, or, based on the intersection of the first video clip set and the second video clip set, aggregation can be performed to obtain a video clip set including specific content.

[0212] In addition, after the intersection of the first video segment set and the second video segment set is taken, a video segment set can be constructed based on the time-continuous video segments determined after the intersection of the video segments. In this case, one video segment may correspond to the intersection of multiple first video segments and second video segments.

[0213] In this way, with the help of the trained neural network model, the business object corresponding to the voice can be quickly and effectively identified based on the target voice features of each business object constructed by the model and the matching of the voice features in the video clip to be identified.

[0214] Furthermore, in a specific video playback scenario, the processing device can respond to the content playback indication triggered by the target object for the specified business object in the data to be processed, determine the target data content associated with the specified business object, so that the video content related to the specified business object can be played on the playback device of the subsequent target object.

[0215] Specifically, the processing device can respond to the target object on the associated playback device, and obtain the target data content associated with the specified business object for the content display request triggered by the specified business object in the data to be processed; and then send the target data content to the playback device so that the playback device plays the video data corresponding to the target data content.

[0216] It should be noted that, for the content display request triggered by the target object, it can be triggered when the target object plays the video content corresponding to the data to be processed, or it can be triggered before the target object plays the video content corresponding to the data to be processed. This application does not impose any specific restrictions on this.

[0217] In addition, considering that the target data content may specifically correspond to a time aggregation result, or a set of video clips; then, in the case where the target data content is a time aggregation result, the corresponding playback range can be marked on the time axis based on the time aggregation result, so that when the playback device associated with the target object plays the video data, in the absence of active intervention by the target object, the video data within the corresponding time range can be played in chronological order;

[0218] In the case where the target data content is a video clip set, the video clip set can be directly sent to the playback device associated with the target object, so that when the playback device plays the video data, in the case of no active intervention by the target object, the video clips associated with the time information are played in chronological order. For example, if the time information corresponding to a video clip is 00:05:10,000-00:07:15,300, the corresponding time information can be identified when playing the video clip.

[0219] The following is a description of the relevant processing process by taking content aggregation of roles played by different stars in film and television dramas as an example in conjunction with the accompanying drawings:

[0220] See attached Figure 3 As shown, it is a schematic diagram of the process of content aggregation for film and television dramas in an embodiment of the present application. Figure 3 It can be seen from the schematic content that during the process of content aggregation, the processing can be divided into modules within the processing device.

[0221] In the processing of the video sentence segmentation module, considering that film and television works are usually very long, for example, movies are usually 1-2 hours long, and TV series usually have dozens of episodes, each of which is dozens of minutes long. When constructing the target speech features of stars, sub-videos can be selected from the long video, and then after the target speech features are generated based on the sub-videos, content aggregation is performed in the long video, wherein the sub-videos include multiple speaking clips of the roles played by each star that need to be targeted for content aggregation.

[0222] For example, a long video may be the content of dozens of episodes in a TV series, and the extracted sub-videos may be a few episodes of the TV series selected from it.

[0223] A complete long video source medium usually contains two parts of information. One part is the video image and audio, and the other part is the subtitle information in the video. The commonly used subtitle file format is srt, where the subtitle srt file contains the start and end time of each sentence in the video and the text content of the video.

[0224] In the processing process corresponding to the video sentence segmentation module, the processing device segments the sub-video into multiple segments based on the start and end time of each line, and each segment contains a line in the video. Finally, each segment contains three pieces of information: video screen segment, lines, and voice.

[0225] In the corresponding processing process of the video character recognition module, for each of the above-mentioned segmented video clips, the celebrity character appearing in the current picture of the video is identified.

[0226] Specifically, face region detection and face feature vector extraction can be performed successively to extract the face feature vector in each image frame, and then the celebrities who can be successfully matched can be identified based on a pre-established face library containing multiple face feature vectors. The face library can be a pre-collected batch of celebrities and their corresponding pictures (called reference pictures), and the face feature vectors of the reference pictures are extracted based on face detection (MTCNN) and face recognition (ArcFace), and then created according to the correspondence between the celebrities and the face feature vectors.

[0227] In the corresponding processing process of the speech feature extraction module, for each segmented video segment, the MFCC features corresponding to each speech frame in the speech segment are extracted.

[0228] In the processing corresponding to the celebrity character recognition model that is not in the database, considering that in the processing of the video character recognition module, after calculating the similarity between the extracted facial feature vector and all the facial feature vectors in the face library, there may be some personal facial feature vectors that are not similar to all the feature vectors in the face library or the similarity does not exceed a given threshold, which will result in no recognition results for the corresponding faces.

[0229] There are usually two reasons why the video character recognition module cannot obtain face recognition results: the celebrity character is covered in the face database, but due to factors such as facial posture, expression, lighting, makeup, etc., the similarity between the image and the feature vector in the face database is not high, resulting in no output; the celebrity character is not covered in the face database, resulting in the inability to identify the celebrity character.

[0230] In the technical solution proposed in this application, the face recognition results can be supplemented by analyzing the voice features as a whole. Although it is affected by factors such as facial posture, expression, lighting, makeup, etc. and not detected at the image level, as long as there is a voice dialogue of the celebrity, it can be supplemented through voice recognition.

[0231] In the face of the problem that the celebrity is not covered in the face database, resulting in the inability to identify the celebrity, in the processing corresponding to the celebrity recognition model not in the database, for all video clips with face detection but no face recognition in the video character recognition module (that is, the face can be recognized but the specific face cannot be determined), the MFCC features of the voice corresponding to each video clip can be obtained through the voice feature extraction module. Since these voices may correspond to multiple celebrities, the audio feature clustering algorithm (DBSCAN) is used to cluster the MFCC features into multiple clusters, each cluster corresponding to a celebrity. Then, each cluster is sent to the operation or outsourcing for labeling. If it is a new celebrity, the correspondence between the celebrity and the face feature vector is stored in the face database, so that the celebrity can be identified in the subsequent process.

[0232] In the corresponding processing process of the data self-labeling module, based on the results of the above-mentioned character recognition module and speech feature extraction module, speech data sets (i.e., speech frame sets) of different stars are automatically collected for generating training samples when the target speech features of each star are constructed using a neural network model.

[0233] In the processing corresponding to the speaker recognition model training module, training samples are generated according to the collected speech of each celebrity, and the deep learning-based speaker recognition model is trained based on the training samples.

[0234] Furthermore, in the speaker recognition prediction and aggregation stage, the trained speaker recognition model is used to perform speaker recognition on the voice in each video clip in the long video. Finally, the identified clips for each star are summarized to obtain the "only watch him / her" recognition result of the star based on speaker recognition, where "only watch him / her" refers to a processing method constructed to meet the viewing needs: only watch content related to a specified star.

[0235] This result is combined with the "only watch Ta" result obtained from the image content level, that is, the video clips identified by both are taken as the union, and finally returned to the user.

[0236] In this way, from the user's viewing perspective, since film and television works revolve around celebrities, users can use the "Only Watch Him" ​​function when watching TV series or variety shows to select their favorite celebrities and automatically jump to play all clips containing the celebrity in each episode, which improves the user's viewing experience. In addition, the emergence of the "Only Watch Him" ​​function can help users determine the celebrity's performance clips and understand the appearance time of the celebrity in a single episode of the film and television works, thereby meeting personalized viewing needs.

[0237] It should be noted that in the embodiments of the present application, considering that there are not many cases in which background music is played while speaking in film and television dramas, and the speech frame set can be understood as extracted from the speech content of the film and television drama, therefore, the influence of possible background music on celebrity voice recognition can be ignored.

[0238] See also Figure 4 As shown, it is a schematic diagram of the content aggregation effect in the embodiment of the present application. Figure 4 From the description, it can be seen that the video between the time 00:05:10,000-00:05:15,300 includes voice content and image content, the video between the time 00:05:15,300-00:05:20,30 includes only image content, and the video between the time 00:05:20,30-00:05:25:130 includes image content and voice content without human faces. In the case where both lines are spoken by "Role 1 played by Star 1", by adopting the technical solution proposed in this application, the videos with a playback time of 00:05:10,000-00:05:25:130 will be aggregated into the data content corresponding to Role 1 played by Star 1.

[0239] Based on the above processing process, it can be seen that in the technical solution proposed in this application, first, the sub-video is divided into several video clips according to the lines through the film and television sentence segmentation module; secondly, the celebrity character recognition is performed based on the picture information of each video clip through the video character recognition module, and the MFCC features are extracted based on the voice information of each video clip through the voice feature extraction module; for video clips with face detection but no face recognition results, the celebrity character recognition module that has not been stored in the database first performs MFCC feature clustering, and then undergoes manual verification to dig out new celebrity characters that may not be stored in the database; the video clips are summarized according to the celebrity characters that appear through the data self-labeling module, and then the noise data is removed through the anomaly detection algorithm to obtain the voice frame set of each celebrity as training data; then the speaker recognition model is trained based on the above training data through the speaker recognition model training module; finally, the speaker recognition is performed on the voice of each video clip through the speaker recognition prediction and summary module, and the results of each celebrity are summarized from the voice level.

[0240] In this way, based on the content aggregation method proposed in the present application, even if a certain star is not included in the video screen, all the speaking clips of the star can be found based on the speaker recognition information. Therefore, for each star, under the business requirement of "only watching TA" (that is, only watching the video content related to the role played by a specified star in the film and television drama), more complete video content can be aggregated, and content aggregation will not be missing because the character is speaking but not in the video screen.

[0241] Moreover, compared with the existing voice recognition methods, this application has significant progress. If the conventional voice recognition method is used for recognition, it is necessary to register the voice in advance, that is, upload the audio of each star in advance, extract the voice features, and store them in the registered voice library. Then, for the content of the film and television drama, the features are extracted and compared with the features in the registered voice library for similarity, and the most similar features are found to predict the star. However, the conventional approach has several limitations: manual uploading is required; the registered voice library will be missing for niche or new stars, resulting in unrecognition; some dramas have dubbing by others, resulting in the voice in the drama not matching the voice of the star himself, which will lead to unrecognition.

[0242] In contrast, this application does not require pre-registration of voices. Instead, it extracts traditional MFCC voice features from the content of film and television dramas based on face recognition of celebrity information, and then performs anomaly detection on each celebrity's voice to remove noise. After that, the celebrity information and voice can be matched to establish a voice frame set corresponding to the celebrity. The advantages of this approach are: no manual upload is required; for niche or new and popular celebrities, as long as there are celebrity face recognition results, they can be added to the voice library, and then voice recognition can be performed; some dramas are dubbed by others, but since this application extracts from the content of the film and television drama itself, it can also be recognized even if dubbed by others.

[0243] In addition, based on the technical solution proposed in this application, when new dramas are put on the shelves, celebrity information that has not been stored in the database can be collected in combination with image and voice information, and then added to the celebrity face database after manual verification by operations or annotations. Therefore, the range of the recognizable celebrity face database can be dynamically expanded, thereby increasing the coverage of celebrities by the "Only See Him" ​​function.

[0244] Based on the same inventive concept, refer to Figure 5 As shown, it is a schematic diagram of the logical structure of the content aggregation device in the embodiment of the present application. The content aggregation device 500 includes an acquisition unit 501, an extraction unit 502, a clustering unit 503, and an aggregation unit 504, wherein:

[0245] The acquisition unit 501 is used to acquire sub-data containing each business object from the data to be processed; the sub-data includes: a sub-video composed of a sub-image content and a sub-voice content, and a sub-caption set associated with the sub-video;

[0246] The extraction unit 502 is used to divide the sub-video into sub-video segments based on the time range associated with each subtitle in the sub-subtitle set, and perform the following operations for each sub-video segment: extracting each business object feature based on each corresponding image frame, and extracting each voice frame feature based on each corresponding voice frame;

[0247] The clustering unit 503 is used to obtain the speech frame set associated with each business object according to the matching between the business object features corresponding to each sub-video segment and the business object features of each pre-stored historical business object, combined with the clustering result of the speech frame features in each speech frame corresponding to each sub-video segment;

[0248] Aggregation unit 504 is used to obtain target speech features corresponding to each business object based on the speech frame sets associated with each business object, and aggregate the data content associated with each business object in the data to be processed according to the business object features and target speech features corresponding to each business object.

[0249] Optionally, when the speech frame sets associated with each service object are obtained by combining the clustering results of the speech frame features in each speech frame corresponding to each sub-video segment, the clustering unit 503 is used to:

[0250] In each sub-video segment, determining a target sub-video segment of a known corresponding business object and other sub-video segments of an unknown corresponding business object;

[0251] Based on the target sub-video segment, a speech frame set associated with the corresponding business object is constructed, and the features of each speech frame corresponding to other sub-video segments are clustered to obtain a speech frame set marked with the associated business object.

[0252] Optionally, based on the target sub-video segment, a speech frame set associated with a corresponding service object is constructed, and the clustering unit 503 is further used to:

[0253] For each business object corresponding to the target sub-video segment, perform the following operations:

[0254] Determine each speech frame included in the corresponding target sub-video segment and the speech frame features corresponding to each speech frame, and use a preset anomaly detection algorithm to detect the features of each speech frame to determine an abnormal speech frame in each speech frame;

[0255] Based on each speech frame from which the abnormal speech frame is deleted, a set of associated speech frames is constructed.

[0256] Optionally, when clustering the features of each speech frame corresponding to other sub-video segments to obtain a speech frame set marked with an associated business object, the clustering unit 503 is used to:

[0257] For each speech frame feature corresponding to other sub-video segments, a preset clustering algorithm is used to cluster the speech frame features whose similarity meets the set conditions to obtain a feature set of each speech frame;

[0258] Each speech frame feature set obtained based on clustering triggers a labeling request, obtains a business object labeled for each speech frame feature set, and obtains each speech frame set labeled with related business objects based on the correspondence between the speech frame features in each speech frame feature set and the speech frames.

[0259] Optionally, when determining a target sub-video segment of a known corresponding business object, the clustering unit 503 is configured to:

[0260] For each sub-video clip, do the following:

[0261] When there are target business object features that successfully match the business object features associated with the sub-video clips in the pre-stored business object features of the historical business objects, determining the target historical business objects corresponding to the target business object features;

[0262] When it is determined that each target historical service object corresponds to the same historical service object, the sub-video segment is determined to be a target sub-video segment of a known corresponding service object.

[0263] Optionally, when it is determined that each target historical business object corresponds to a different historical business object, the clustering unit 503 is configured to perform any one of the following operations:

[0264] Determine the sub-video segment as another sub-video segment of an unknown corresponding business object;

[0265] An object annotation request is triggered based on the sub-video segment, and based on the acquired business object annotation result, a designated business object corresponding to the sub-video segment is determined, and the sub-video segment is determined as a target sub-video segment corresponding to the designated business object.

[0266] Optionally, when determining other sub-video segments of unknown corresponding business objects, the clustering unit 503 is used to:

[0267] For each sub-video clip, do the following:

[0268] When there is no target business object feature successfully matched with each business object feature associated with the sub-video segment in the pre-stored business object features of each historical business object, the sub-video segment is determined to be another sub-video segment of an unknown corresponding business object.

[0269] Optionally, the data to be processed includes: a video consisting of image content and voice content, and a subtitle set associated with the video; according to the business object features and target voice features corresponding to each business object, when aggregating the data content associated with each business object in the data to be processed, the aggregation unit 504 is used to:

[0270] Based on the business object features of each business object, object matching is performed in the image content, and for each business object, aggregating to obtain a corresponding first video clip set;

[0271] The speech content of the data to be processed is divided into speech frames, and based on the time range corresponding to each speech frame, the video is divided into second video segments, and for each speech frame, the corresponding speech frame features are extracted respectively, and according to the matching conditions between each speech frame feature and each target speech feature, a second video segment set corresponding to each business object is obtained by aggregation;

[0272] For each business object, based on the corresponding first video segment set and second video segment set, data content aggregated for the business object is obtained.

[0273] Optionally, when extracting features of each business object based on each corresponding image frame, the extraction unit 502 is used to:

[0274] Determine each image frame included in the corresponding sub-image content, and use a preset region detection algorithm to detect a business object region containing a business object in each image frame;

[0275] A preset feature extraction algorithm is used to extract features from the business object area in each image frame to obtain corresponding business object features.

[0276] Optionally, when extracting features of each speech frame based on the corresponding speech frames, the extraction unit 502 is used to:

[0277] Splitting the sub-speech content segment corresponding to the sub-video segment into individual speech frames, and performing windowing processing on each speech frame obtained by the splitting to obtain each processed speech frame;

[0278] For each processed speech frame, a preset speech feature extraction technology is used to extract the corresponding Mel-frequency cepstral coefficient MFCC features.

[0279] Optionally, when obtaining the target speech features corresponding to each business object based on the speech frame sets associated with each business object, the aggregation unit 504 is configured to include:

[0280] Based on the speech frame sets associated with each business object, construct each training sample, wherein a training sample includes: a sample speech constructed from the speech frames, and a labeled sample object;

[0281] Based on each training sample, the preset object speech recognition model is trained for multiple rounds of iterations to obtain a trained target object speech recognition model, and obtain target speech features constructed for each business object within the target object speech recognition model.

[0282] Optionally, after aggregating the data contents associated with each business object in the data to be processed, the device further includes a response unit 505, which is used to:

[0283] In response to a content display request triggered by a specified business object in the data to be processed on an associated playback device, the target object obtains target data content associated with the specified business object;

[0284] The target data content is sent to the playback device so that the playback device plays the video data corresponding to the target data content.

[0285] After introducing the content aggregation method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0286] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0287] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application, wherein the electronic device specifically corresponds to Figure 1 The processing device 110 shown in FIG. Figure 6 As shown, it is a schematic diagram of a hardware structure of an electronic device using an embodiment of the present application, and the electronic device 600 may include at least a processor 601 and a memory 602. Among them, the memory 602 stores a computer program, and when the computer program is executed by the processor 601, the processor 601 performs any of the above steps of content aggregation.

[0288] In some possible implementations, the electronic device of the present application may include at least one processor and at least one memory. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of content aggregation according to various exemplary implementations of the present application described above in this specification. For example, the processor may perform the following steps: Figure 2A Follow the steps shown in .

[0289] Refer to the following Figure 7 The electronic device 700 according to this embodiment of the present application is described. Figure 7 As shown, it is a schematic diagram of a hardware structure of another electronic device in an embodiment of the present application, and the electronic device 700 specifically corresponds to Figure 1 The client device 120 is shown. The components of the electronic device 700 may include but are not limited to: the at least one processing unit 701, the at least one storage unit 702, and a bus 703 connecting different system components (including the storage unit 702 and the processing unit 701).

[0290] Bus 703 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0291] The storage unit 702 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 7021 and / or a cache memory 7022 , and may further include a read-only memory (ROM) 7023 .

[0292] The storage unit 702 may also include a program / utility 7025 having a set (at least one) of program modules 7024, such program modules 7024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0293] The electronic device 700 may also communicate with one or more external devices 704 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable objects to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 705. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 706. As shown, the network adapter 706 communicates with other modules for the electronic device 700 via a bus 703. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0294] Based on the same inventive concept as the above method embodiment, various aspects of content aggregation provided by the present application can also be implemented in the form of a computer program product, which includes a computer program. Figure 6 or Figure 7 When the computer program is run on the electronic device shown in the figure, the computer program is used to make the electronic device execute the steps of the content aggregation method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device can execute the following steps: Figure 2A Follow the steps shown in .

[0295] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0296] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0297] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A content aggregation method, characterized in that: include: Obtaining sub-data containing each business object from the data to be processed; the sub-data includes: a sub-video composed of sub-image content and sub-voice content, and a sub-caption set associated with the sub-video; Based on the time range associated with each subtitle in the subtitle set, the subvideo is divided into subvideo segments, and for each subvideo segment, the following operations are performed: based on the corresponding image frames, the features of each business object are extracted, and based on the corresponding voice frames, the features of each voice frame are extracted; According to the matching between the business object features corresponding to each of the sub-video clips and the business object features of each pre-stored historical business object, combined with the clustering results of the speech frame features in each of the speech frames corresponding to each of the sub-video clips, the speech frame sets associated with each of the business objects are obtained; Based on the speech frame sets associated with each of the business objects, the target speech features corresponding to each of the business objects are obtained, and according to the business object features and target speech features corresponding to each of the business objects, the data contents associated with each of the business objects are aggregated in the data to be processed.

2. The method according to claim 1, characterized in that The step of combining the clustering results of the speech frame features in the speech frames corresponding to the sub-video clips to obtain the speech frame sets associated with the business objects includes: Determine, in each of the sub-video segments, a target sub-video segment of a known corresponding business object and other sub-video segments of an unknown corresponding business object; Based on the target sub-video segment, a speech frame set associated with a corresponding business object is constructed, and the speech frame features corresponding to the other sub-video segments are clustered to obtain a speech frame set marked with the associated business object.

3. The method according to claim 2, characterized in that The step of constructing a speech frame set associated with a corresponding service object based on the target sub-video segment further includes: For each business object corresponding to the target sub-video segment, perform the following operations: Determine each speech frame included in the corresponding target sub-video segment, and speech frame features corresponding to each of the speech frames, and use a preset anomaly detection algorithm to determine an abnormal speech frame among the speech frames by detecting the features of each speech frame; Based on deleting the speech frames of the abnormal speech frame, a set of associated speech frames is constructed.

4. The method according to claim 2, characterized in that The clustering of the speech frame features corresponding to the other sub-video segments to obtain a speech frame set marked with the associated business objects includes: For each speech frame feature corresponding to the other sub-video segments, a preset clustering algorithm is used to cluster the speech frame features whose similarity meets the set conditions to obtain a feature set of each speech frame; Each speech frame feature set obtained based on clustering triggers a labeling request to obtain business objects labeled respectively for the said speech frame feature sets, and according to the correspondence between the speech frame features in the said speech frame feature sets and the speech frames, each speech frame set labeled with related business objects is obtained.

5. The method according to claim 2, characterized in that The step of determining a target sub-video segment of a known corresponding business object includes: For each sub-video clip, do the following: When there are target business object features that are successfully matched with the business object features associated with the sub-video clips among the pre-stored business object features of the historical business objects, determining the target historical business objects corresponding to the target business object features; When it is determined that each target historical service object corresponds to the same historical service object, the sub-video segment is determined to be a target sub-video segment of a known corresponding service object.

6. The method according to claim 5, characterized in that When determining that each target historical business object corresponds to a different historical business object, perform any of the following operations: Determining the sub-video segment as another sub-video segment of an unknown corresponding business object; An object labeling request is triggered based on the sub-video segment, and based on the obtained business object labeling result, a designated business object corresponding to the sub-video segment is determined, and the sub-video segment is determined as a target sub-video segment corresponding to the designated business object.

7. The method according to claim 2, characterized in that Determine other sub-video segments of the unknown corresponding business object, including: For each sub-video clip, do the following: When there is no target business object feature successfully matched with each business object feature associated with the sub-video segment in the pre-stored business object features of each historical business object, the sub-video segment is determined to be another sub-video segment of an unknown corresponding business object.

8. The method according to claim 1, characterized in that The data to be processed includes: a video consisting of image content and voice content, and a subtitle set associated with the video; and according to the business object features and target voice features corresponding to each of the business objects, aggregating the data content associated with each of the business objects in the data to be processed includes: Based on the business object features of each business object, object matching is performed in the image content, and for each business object, aggregating to obtain a corresponding first video clip set; The speech content of the data to be processed is divided into speech frames, and based on the time range corresponding to each of the speech frames, the video is divided into second video segments, and for each of the speech frames, corresponding speech frame features are extracted respectively, and according to the matching conditions between each of the speech frame features and each of the target speech features, aggregating to obtain a second video segment set corresponding to each of the business objects; For each business object, based on the corresponding first video segment set and second video segment set, data content aggregated for the business object is obtained.

9. The method according to any one of claims 1 to 8, characterized in that The extracting each business object feature based on each corresponding image frame includes: Determine each image frame included in the corresponding sub-image content, and use a preset region detection algorithm to detect a business object region containing a business object in each image frame; A preset feature extraction algorithm is used to extract features from the business object regions in each image frame to obtain corresponding business object features.

10. The method according to any one of claims 1 to 8, characterized in that The extracting features of each speech frame based on the corresponding speech frames includes: Splitting the sub-voice content segment corresponding to the sub-video segment into individual voice frames, and performing windowing processing on each of the split voice frames to obtain processed voice frames; For each processed speech frame, a preset speech feature extraction technology is used to extract the corresponding Mel-frequency cepstral coefficient MFCC feature.

11. The method according to any one of claims 1 to 8, characterized in that: The obtaining, based on the speech frame sets associated with the business objects, the target speech features corresponding to the business objects respectively, comprises: Based on the speech frame sets associated with the business objects, constructing training samples, wherein a training sample includes: sample speech constructed from speech frames, and annotated sample objects; Based on the training samples, the preset object speech recognition model is trained for multiple rounds of iterations to obtain a trained target object speech recognition model, and obtain target speech features constructed for each business object within the target object speech recognition model.

12. The method according to any one of claims 1 to 8, characterized in that: After aggregating the data contents associated with each of the business objects in the data to be processed, the method further includes: In response to a content display request triggered by a specified business object in the data to be processed on an associated playback device, the target data content associated with the specified business object is obtained; The target data content is sent to the playback device, so that the playback device plays the video data corresponding to the target data content.

13. A content aggregation device, characterized in that: include: An acquisition unit, used to acquire sub-data containing each business object from the data to be processed; the sub-data includes: a sub-video composed of sub-image content and sub-voice content, and a sub-caption set associated with the sub-video; The extraction unit is used to divide the sub-video into sub-video segments based on the time range associated with each subtitle in the sub-subtitle set, and for each sub-video segment, perform the following operations: extracting each business object feature based on each corresponding image frame, and extracting each voice frame feature based on each corresponding voice frame; A clustering unit, configured to obtain a speech frame set associated with each of the business objects according to a matching condition between the business object features corresponding to each of the sub-video clips and the business object features of each pre-stored historical business object, combined with a clustering result of speech frame features in each of the speech frames corresponding to each of the sub-video clips; An aggregation unit is used to obtain the target speech features corresponding to each business object based on the speech frame sets associated with each business object, and aggregate the data content associated with each business object in the data to be processed according to the business object features and target speech features corresponding to each business object.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images