Video processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a video processing method, device, electronic device and computer-readable storage medium, which relate to the fields of artificial intelligence and audio and video processing technology. The method includes: obtaining a first image set, a second image set and a reference video frame set for at least part of the video frames in the target video; determining the correspondence between the first image and the second image belonging to the same target object; obtaining at least one first image cluster and a second image cluster, and setting an object label for each first image cluster; for each second image cluster, determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster; and determining the target video segment corresponding to each object label from the target video. In the embodiment of the present application, when the first part is not visible, the second part provides additional guidance information, which improves the accuracy and recall rate of obtaining task highlights.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and audio and video processing technology. Specifically, the present application relates to a video processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Character highlights can be applied to application areas such as retrieval, recommendation and secondary production, and are widely used in industries such as radio and television, and online audio-visual.

[0003] Related technologies mostly use unimodal features of the face to make judgments. This is because, with the support of deep learning technology, face detection, face feature extraction, and face classification / clustering and other facial visual analysis fields have made great progress. The precision and recall rate in some scenarios can even surpass the performance of manual labeling.

[0004] However, in actual applications, the characters in the video have various postures, lighting and deformation conditions. Once the face is blocked, the character is facing away from the camera, or the character is lowering his head, face detection will fail, resulting in missed detections. Summary of the Invention

[0005] The embodiments of the present application provide a video processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can solve the above-mentioned problems of the prior art. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a video processing method is provided, the method comprising:

[0007] Detecting a first portion and a second portion of a target object on at least a portion of video frames in a target video to obtain a first image set, a second image set, and a reference video frame set; each first image in the first image set is a subregion of the first portion of a target object in a video frame; each second image in the second image set is a subregion of the second portion of a target object in a video frame; and each reference video frame in the reference video frame set contains at least one first image and at least one second image.

[0008] For each reference video frame, determining a correspondence between a first image and a second image belonging to the same target object;

[0009] Clustering the images in the first image set to obtain at least one first image cluster, clustering the images in the second image set to obtain at least one second image cluster, and setting an object label for each first image cluster;

[0010] For each second image cluster, determining an object label of the second image cluster according to an object label of the first image corresponding to a matching image in the second image cluster; the matching image is a second image in the second image cluster having a corresponding relationship;

[0011] According to the object labels corresponding to the first image clusters and the second image clusters whose object labels have been determined, target video segments corresponding to the respective object labels are determined from the target video.

[0012] According to another aspect of an embodiment of the present application, a video processing method is provided, including:

[0013] Detecting a first part and a second part of a target object on at least a portion of video frames in a target video to obtain a first image set and a second image set; each first image in the first image set is a subregion of the first part of a target object in a video frame; and each second image in the second image set is a subregion of the second part of a target object in a video frame;

[0014] The audio corresponding to the target video is divided into multiple audio segments based on characters, and the multiple audio segments are clustered to obtain at least one audio segment cluster; the images in the first image set are clustered to obtain at least one first image cluster, and the images in the second image set are clustered to obtain at least one second image cluster, and an object label is assigned to each of the first image clusters;

[0015] For each audio segment cluster, determining a matching degree between the audio segment cluster and each to-be-matched image cluster, wherein the to-be-matched image cluster includes at least one of the first image cluster or the second image cluster;

[0016] For any first image cluster, an audio clip cluster whose matching degree with the first image cluster meets a second preset condition is used as a reference audio cluster, a second image cluster whose matching degree with the reference audio cluster meets the second preset condition is used as a reference second image cluster, and a target video frame corresponding to the reference audio cluster and an object label of the reference second image cluster are determined as the object label of the first image cluster;

[0017] According to each first image cluster, the second image cluster with determined object labels, and the object labels corresponding to the target video frame, a target video segment corresponding to each object label is determined from the target video.

[0018] According to another aspect of an embodiment of the present application, a video processing device is provided, the device comprising:

[0019] A multi-mode detection module is configured to detect a first portion and a second portion of a target object in at least a portion of video frames in a target video, thereby obtaining a first image set, a second image set, and a reference video frame set; each first image in the first image set is a subregion of the first portion of a target object in a video frame; each second image in the second image set is a subregion of the second portion of a target object in a video frame; and each reference video frame in the reference video frame set contains both at least one first image and at least one second image.

[0020] a correspondence determination module, configured to determine, for each reference video frame, a correspondence between a first image and a second image belonging to the same target object;

[0021] a clustering module, configured to cluster the images in the first image set to obtain at least one first image cluster, cluster the images in the second image set to obtain at least one second image cluster, and set an object label for each first image cluster;

[0022] a label determination module configured to determine, for each second image cluster, an object label of the second image cluster based on an object label of a first image corresponding to a matching image in the second image cluster; the matching image being a second image in the second image cluster having a corresponding relationship;

[0023] The video highlights module is used to determine target video segments corresponding to each object label from the target video according to the object labels corresponding to each first image cluster and the second image cluster with determined object labels.

[0024] As an optional method, the multimode detection module is also used to:

[0025] The audio corresponding to the target video is divided into multiple audio segments based on characters, and the multiple audio segments are clustered to obtain at least one audio segment cluster;

[0026] For each audio segment cluster, determining a matching degree between the audio segment cluster and each to-be-matched image cluster, where the to-be-matched image cluster includes at least one of the first image cluster and the second image cluster;

[0027] The tag determination module is specifically used to:

[0028] For each second image cluster, the object label of the second image cluster is determined according to the object label of the first image corresponding to the matching image in the second image cluster and the matching degree between the second image cluster and each audio segment cluster.

[0029] As an optional method, the device further includes:

[0030] a shot segmentation module, configured to segment a target video into a plurality of video segments based on a shot, and determine a preset number of video frames from each video segment as at least a portion of the video frames;

[0031] The video highlights module is specifically used for:

[0032] For each object label, the first image cluster and the second image cluster with the object label are both used as target image clusters, and the video segment corresponding to each image in the target image cluster is used as the target video segment corresponding to the object label.

[0033] As an optional method, the label determination module is specifically used to: if the number of matching images in the second image cluster and the discrete degree of the object labels of the first image corresponding to the matching image meet the first preset condition, then the object label with the largest number among the object labels of all matching images is used as the object label of the second image cluster.

[0034] As an optional method, the tag determination module is specifically used to:

[0035] If at least one of the number of matching images in the second image cluster or the degree of discreteness of the object labels of the first images corresponding to the matching images does not meet the first preset condition, the object label of the second image cluster is determined according to the degree of matching between each audio clip cluster and each image cluster to be matched.

[0036] As an optional method, the tag determination module is specifically used to:

[0037] If the matching degree between the second image cluster and a first image cluster and the same audio clip cluster meets a second preset condition, the object label corresponding to the first image cluster is used as the object label corresponding to the second image cluster.

[0038] As an optional method, the device further includes:

[0039] The image moving module is used to determine, for each matching image in the second image cluster, a target object label of the first image corresponding to the matching image, and move the matching image to a target second image cluster having the target object label.

[0040] As an optional method, the tag determination module is specifically used to:

[0041] If the degree of matching between the second image cluster and all audio clip clusters does not meet the second preset condition, then for each non-matching image in the second image cluster, determine the similarity between the non-matching image and the cluster center of each second image cluster; if the difference between the maximum similarity and the second maximum similarity is greater than a preset threshold, move the non-matching image to the second image cluster with the maximum similarity;

[0042] The non-matching image is a second image in the second image cluster that has no corresponding relationship.

[0043] As an optional method, the shot segmentation module includes:

[0044] Confidence unit, used to predict the segmentation confidence of each video frame through the pre-trained shot segmentation model;

[0045] A sliding window unit is used to count the number of video frames whose segmentation confidence is higher than a segmentation threshold within a plurality of video frames through a sliding window. If the number exceeds a preset value, the video frame with the highest segmentation confidence in the sliding window is used as the video segmentation point;

[0046] The segmentation module is used to segment the target video according to the video segmentation points.

[0047] As an optional method, the clustering module includes:

[0048] an initial feature unit, configured to obtain a first feature of each first image and an initial second feature of each second image;

[0049] a neighboring image pair unit, configured to determine each nearest neighboring first image pair based on the first feature of each first image, and to determine each nearest neighboring second image pair based on the initial second feature of each second image, wherein the two first images in each nearest neighboring first image pair are nearest neighbors to each other, and the two second images in each nearest neighboring second image pair are nearest neighbors to each other;

[0050] an image pair updating unit, configured to update, based on the correspondence between the two first images in each of the most adjacent first image pairs, the most adjacent second image pairs where the second images corresponding to the two first images are located;

[0051] a feature updating unit configured to, for each second image, use a nearest image in all updated nearest-neighbor second image pairs of the second image as a target nearest-neighbor image, and obtain a target second feature of the second image based on an initial second feature of the target nearest-neighbor image;

[0052] The clustering unit is configured to cluster the first features of all first images to obtain at least one first image cluster; and cluster the target second features of all second images to obtain at least one second image cluster.

[0053] As an optional method, the image pair updating unit is specifically used to:

[0054] For each nearest first image pair, two first images in the nearest first image pair are determined; if both first images have corresponding target second images, the two corresponding target second images are determined as the nearest second image pair.

[0055] According to another aspect of an embodiment of the present application, a video processing device is provided, the device comprising:

[0056] a video frame part detection module configured to detect a first part and a second part of a target object in at least a portion of video frames in a target video, thereby obtaining a first image set and a second image set; wherein each first image in the first image set is a subregion of the first part of a target object in a video frame; and each second image in the second image set is a subregion of the second part of a target object in a video frame;

[0057] a multi-modal clustering module configured to segment the audio corresponding to the target video into a plurality of audio segments based on characters, cluster the plurality of audio segments to obtain at least one audio segment cluster; cluster the images in the first image set to obtain at least one first image cluster, cluster the images in the second image set to obtain at least one second image cluster, and assign an object label to each of the first image clusters;

[0058] an audio matching module, configured to determine, for each audio segment cluster, a degree of matching between the audio segment cluster and each to-be-matched image cluster, wherein the to-be-matched image cluster includes at least one of the first image cluster or the second image cluster;

[0059] a multimodal matching module configured to, for any first image cluster, use an audio segment cluster whose degree of matching with the first image cluster meets a second preset condition as a reference audio cluster, use a second image cluster whose degree of matching with the reference audio cluster meets the second preset condition as a reference second image cluster, and determine a target video frame corresponding to the reference audio cluster and an object label of the reference second image cluster as the object label of the first image cluster;

[0060] The video segment screening module is used to determine target video segments corresponding to respective object tags from the target video based on respective first image clusters, second image clusters with determined object tags, and object tags corresponding to target video frames.

[0061] As an optional embodiment, the video frame part detection module is further configured to:

[0062] Obtaining a reference video frame set, wherein each reference video frame in the reference video frame set simultaneously contains at least one first image and at least one second image;

[0063] The device also includes:

[0064] a correspondence module, configured to determine, for each reference video frame, a correspondence between a first image and a second image belonging to the same target object;

[0065] A second image cluster label determination module is configured to determine, for each second image cluster, an object label of the second image cluster based on an object label of a first image corresponding to a matching image in the second image cluster; the matching image is a second image in the second image cluster having the corresponding relationship.

[0066] According to another aspect of an embodiment of the present application, an electronic device is provided. The electronic device includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0067] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0068] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps of the above method when executed by a processor.

[0069] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0070] By performing multimodal (first and second part) detection on at least some video frames in a target video, a first image set, a second image set, and a reference video frame set are obtained. Intra-modal association is performed on the first image set and the second image set respectively to obtain at least one first image cluster and at least one second image cluster, and an object label is set for each first image cluster. The reference video frame set is used to determine the correspondence between the first image and the second image belonging to the same target object. For each second image cluster, the object label of the second image cluster is determined based on the object label of the first image corresponding to the matching image in the second image cluster, thereby achieving inter-modal association. Based on the object labels corresponding to each first image cluster and the second image cluster with determined object labels, the target video segment corresponding to each object label is determined from the target video. The present application performs intra-modal and inter-modal association on the features of multiple modalities. Based on the high accuracy of the first part detection, the associated second image cluster can also correspond to the object label to a certain extent. When the first part is not visible, the second part provides additional guidance information, thereby improving the accuracy and recall rate of obtaining the task highlights. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0072] Figure 1 A schematic diagram of the implementation environment of the solution provided in the embodiment of the present application;

[0073] Figure 2 A flowchart of a video processing method provided in an embodiment of the present application;

[0074] Figure 3 A schematic diagram of a process for object label recognition based on multimodal features provided in an embodiment of the present application;

[0075] Figure 4 A schematic diagram of a first image and a second image detected when a subject changes multiple sets of clothes, provided in an embodiment of the present application;

[0076] Figure 5 A flowchart of a multimodal nearest neighbor feature optimization method provided in an embodiment of the present application;

[0077] Figure 6 A flowchart of a video processing method provided in an embodiment of the present application;

[0078] Figure 7 A schematic diagram of a process for retrieving character materials provided in an embodiment of the present application;

[0079] Figure 8 A schematic diagram of a process for viewing only video clips of specific characters provided in an embodiment of the present application;

[0080] Figure 9 A flowchart of another video processing method provided in an embodiment of the present application;

[0081] Figure 10 A schematic diagram of the structure of a video processing system provided in an embodiment of the present application;

[0082] Figure 11 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0083] Figure 12 A schematic structural diagram of another video processing device provided in an embodiment of the present application;

[0084] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0086] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0087] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0088] First, several terms involved in this application are introduced and explained:

[0089] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0090] Computer vision technology (Computer Vision, CV), computer vision is a science that studies how to make machines "see", and more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. The technical solutions provided in the embodiments of the present application mainly relate to image recognition, video processing, face recognition and other technologies in computer vision technology. This can be specifically illustrated by the following embodiments.

[0091] In news or variety shows, shots are often the most basic structural unit. In a shot shot from a fixed camera position, the background and characters usually remain unchanged (for moving cameras, this can be further subdivided to ensure this remains the same).

[0092] A character collection is a collection of video frames containing an image of a specific person within a video. In recent years, with the widespread application of deep learning methods in data and image processing, a growing number of technical solutions have emerged. Existing technical solutions can be categorized into frame-by-frame and alternate-frame processing based on frame extraction methods, facial feature-based and multimodal feature-based based on feature types, and classification, retrieval, and clustering strategies based on character frame classification.

[0093] Although there are many solutions for character collection, most of the related technical solutions follow the following ideas and frameworks. Figure 1 As shown in the figure, video frames are first extracted according to a certain strategy. For these frames, a detector and feature extractor based on deep learning methods are used to obtain the coordinate position and feature information of the person. The feature information here includes but is not limited to one or more of facial image features, head image features, and body image features. Based on these image features, different classification strategies are used to generate a collection of character clips. For example, a classifier is trained to classify image features, and the person corresponding to the video frame is determined based on the classification results. The related technology has the following problems:

[0094] First, using a frame-by-frame or alternate-frame extraction method, while this method achieves higher accuracy, also comes with excessive computational overhead. In fact, in common news or variety shows, shots are often the most basic structural unit. In a shot from a fixed camera position, the background and characters typically remain unchanged (for moving cameras, this can be ensured by further subdividing the shot).

[0095] Secondly, most current face collection solutions only utilize unimodal facial features for judgment. This is because deep learning technology has enabled significant progress in facial visual analysis, including face detection, feature extraction, and classification / clustering. In some scenarios, the accuracy and recall rates even surpass those of manual annotation. However, in real-life production, people in videos exhibit a wide variety of postures, lighting, and deformations. Face detection fails when faces are obscured, facing away from the camera, or looking down, leading to missed detections.

[0096] Third, some technologies are currently exploring the use of multimodal features for joint modeling and character classification. For example, supervised solutions combine features from multiple modalities for classification, but this method requires a large amount of manual labeling and is costly. Another example is unsupervised solutions that use a retrieval or clustering approach to group multiple features of the same person together, and then use a query feature to find all the fragments of that person. The difficulty of this type of approach lies in handling the priorities and conflicts between the features of each modality. When the clustering results of each modality conflict, existing solutions often simply use the face clustering result as the standard, or discard the person instance.

[0097] The video processing method, device, electronic device, computer-readable storage medium, and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0098] First, the embodiments of the present application adopt a frame extraction method based on shot segmentation. For example, a shot segmentation model can be used to cut a complete long video into several shots, and then a clear internal image frame (also called an I frame) is selected from each shot to represent the shot. The representative frames of all shots constitute the video frames to be processed. This total number of frames is not only much smaller than the frames to be processed by frame-by-frame or alternate frame extraction methods, but also more effectively preserves the information of the characters appearing in the shot, thereby significantly improving the speed and performance of subsequent processes.

[0099] Secondly, this embodiment of the application selects features from three modalities: facial features, body features, and vocal features. When the face is visible, the facial image features are used as the basis. When the face is not visible, the body features and vocal features can provide additional guidance information to help recall more video clips, so that the effective clips in the character collection are retained as much as possible.

[0100] Furthermore, to address the common issue of people changing clothes in variety shows, human features can cause interference, leading to instances of the same person being classified into multiple clusters. This embodiment of the application proposes a multimodal Top-K nearest neighbor human feature map optimization method, which effectively improves the problem of people changing clothes and enhances the reliability of human image features.

[0101] Third, the embodiment of the present application proposes a new multimodal clustering fusion method, in which the features of the three modalities are first clustered separately. At this time, the information between the modalities will not interact, and the information conflict between the modalities is minimized. In addition, parallel processing can be performed to save time. In the fusion stage, the priority of the face clustering results, the body clustering results, and the voice clustering results decreases in sequence. In the case of missing information of the former, the latter can provide additional guidance information for the former, so that the clustering results of the three modalities can be more effectively integrated together and the introduced errors can be reduced.

[0102] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0103] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application. The implementation environment of the solution can be implemented as a video processing system. The implementation environment of the solution may include: a terminal device 10 and a server 20.

[0104] The video processing system determines, delivers and stores character highlights of each character in the video content through the terminal device 10 and the server 20.

[0105] The terminal device 10 may be an electronic device such as a mobile phone, a tablet computer, a PC (Personal Computer), a wearable device, a vehicle-mounted terminal device, a VR (Virtual Reality) device, and an AR (Augmented Reality) device, and this application does not limit this. A client running a target application may be installed in the terminal device 10. For example, the target application may be a video content playback application or other application with a video content playback function, such as a short video application, a video playback application, a video editing application, a browser application, etc., and this application does not limit this.

[0106] Server 20 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Server 20 can be the backend server of the target application described above, providing backend services to the client of the target application. Server 20 is used to generate character highlights from the video content for the publisher, and to deliver or store the generated character highlights.

[0107] The embodiment of the present application provides a video processing method, such as Figure 2 As shown, the method includes:

[0108] S101 : Detect a first part and a second part of a target object on at least part of video frames in a target video to obtain a first image set, a second image set, and a reference video frame set.

[0109] The target video of the embodiment of the present application can be an advertising video containing people (or called "video advertising", which means an advertisement presented in video form), or it can be an everyday video shot of people, such as personal selfie videos, group videos, etc. It can also be a program video, such as variety show videos, character interview videos, news broadcast videos, etc. This application does not limit the type of target video.

[0110] The present application does not impose any specific limitation on the method of obtaining at least part of the video frames. For example, video frames can be extracted at preset frame intervals. Alternatively, human voice detection can be performed on the audio corresponding to the target video. Whenever a change in the speaker is detected, the frames at the corresponding moment in the target video and before and after the moment are extracted.

[0111] The present application detects the first part and the second part of the target object on the extracted video frames, respectively, where the first part can be a face and the second part can be a human body, thereby obtaining a first image set, a second image set and a reference video frame set.

[0112] Taking the first part as a face as an example, when performing face detection, the embodiments of the present application can adopt a preset neural network model and perform pre-training based on a preset face dataset. In some optional embodiments, the neural network model of the present application can be a Tinaface model, and the face dataset can be a WIDER FACE dataset.

[0113] As an optional embodiment, the embodiment of the present application can perform quality assessment on each detected first part, and filter out low-quality first images that are too small, too dark, too blurred, or highly deformed, so as not to affect the processing effect of the downstream process.

[0114] Each first image in the first image set is a sub-area of the first part of a target object in a video frame. That is to say, when the first parts of several target objects are detected in a video frame, there are several first images in the video frame. For example, if the first parts of three people are detected in a video frame, then three first images can be obtained from the video frame. The original size of each first image is different, which is related to the pixel area occupied by the corresponding first part in the video frame. Generally speaking, the closer the image acquisition device is to the target object when acquiring the image, the larger the proportion of the target object in the video frame.

[0115] Taking the second part as a human body as an example, when performing human body detection, embodiments of the present application can adopt a preset neural network model and perform pre-training based on a preset human body dataset. In some optional embodiments, the neural network model of the present application can be a lightweight object detection model YOLO (You Only Look Once) 5, and the dataset can be the Microsoft Common Objects in Context (MS COCO) dataset.

[0116] In some embodiments, considering that the YOLO5 model may detect human results that are not needed in actual business, the embodiments of the present application perform some complex post-processing on the human detection results to filter out low-quality, incomplete and non-living human images, etc.

[0117] Each second image in the second image set is a sub-region of a second part of a target object in a video frame. When several second parts of target objects are detected in a video frame, there are several second images in the video frame.

[0118] When performing the first part detection and the second part detection, the present application determines that if a first image and a second image exist simultaneously in a video, the video frame is determined to be a reference video frame and is collected into a reference video frame set.

[0119] S102 : For each reference video frame, determine a correspondence between a first image and a second image belonging to the same target object.

[0120] Taking into account the natural connection relationship between the first part and the second part, the embodiment of the present application can refer to the overlap (Intersection over Union, IOU) between each first image and each second image in the video frame. When the overlap between a first image and a second image is high, it is considered that the first image and the second image belong to the same target object.

[0121] Because the scale of the characters in the video varies, the standard IOU is modified in this embodiment of the application. The new IOU is expressed as the ratio of the intersection of the face and body area to the body area. Generally, in the field of image processing, the origin of the two-dimensional coordinate system is the upper left vertex of the (rectangular) image, the positive direction of the horizontal axis is from the upper left vertex to the upper right vertex, and the positive direction of the vertical axis is from the upper left vertex to the lower left vertex.

[0122] If the coordinates of the upper left corner of the first image are expressed as [x1, y1, x2, y2] and the coordinates of the lower right corner of the second image are expressed as [a1, b1, a2, b2], then the IOU can be expressed as

[0123]

[0124] Through the above-mentioned IOU calculation method, when there is at least one first image and at least one second image in a reference video frame, the first image and the second image belonging to the same target object can be obtained. It should be understood that not all first images and second images in a reference video frame necessarily have a corresponding relationship. For example, due to occlusion by other objects or the posture of the target object, a reference video frame may only contain the first part or the second part of the target object.

[0125] S103 , clustering the images in the first image set to obtain at least one first image cluster, clustering the images in the second image set to obtain at least one second image cluster, and setting an object label for each first image cluster.

[0126] In step S103, the embodiment of the present application performs intra-modal correlation on the first image set and the second image set, respectively. (The modality of the embodiment of the present application is also the detection method of different parts of the target object.) Intra-modal correlation is to sort out information of the same modality. Specifically, an unsupervised clustering method is used to cluster the images in the first image set and the images in the second image set, respectively, to obtain at least one first image cluster and at least one second image cluster. The first images in each first image cluster are similar to each other, and the second images in each second image cluster are similar to each other.

[0127] Specifically, the embodiment of the present application can extract features from each image in the first image set and the second image set through a pre-trained feature extraction model, and then perform clustering using the extracted image features. The embodiment of the present application does not specifically limit the feature extraction model for feature extraction. For example, for the first image, the features of the first image can be extracted based on the SENet50 model pre-trained on the MS-Celeb-1M face dataset. Furthermore, after the SENet50 model is pre-trained, it can be fine-tuned using the VGGFace2 face dataset to obtain a first feature extraction model with more accurate feature extraction. For the second image, the features of the second image can be extracted based on the ResNet50 model pre-trained on the large-scale pedestrian dataset LaST.

[0128] This application does not specifically limit the clustering method; for example, K-Means, density clustering, spectral clustering, hierarchical clustering, etc., are all applicable to this application. After clustering the first image clusters, the embodiment of this application sets an object label for each first image cluster. Due to the high detection accuracy of the first portion, after clustering, it can be assumed that each first image cluster corresponds to a target object. After setting an object label for each first image cluster, all images in the first image cluster also have the object label.

[0129] The object labels of this application may be used solely to distinguish different first image clusters. In some embodiments, they may also be used to represent the identity information of the target object. In other words, the object labels may be Arabic numerals or any distinguishing symbols, or they may be the name or other identity information of the target object. This application does not impose any specific limitations on these terms.

[0130] S104 , for each second image cluster, determining an object label of the second image cluster according to an object label of a first image corresponding to a matching image in the second image cluster; the matching image is a second image in the second image cluster having a corresponding relationship.

[0131] After performing the above-mentioned intra-modal association, the correspondence between the first image and the second image belonging to the same target object can be combined to establish an association between the modalities, that is, to establish an association between the first layer cluster and the second image cluster. Specifically, by determining the correspondence between the first image and the second image in the previous step, for each second image cluster, the second image with a corresponding relationship in the second image cluster can be determined, which is referred to as a matching image in the embodiment of the present application. For each matching image in a second image cluster, the object label of the corresponding first image (in the first image cluster) can be determined, and the object label of the second image cluster can be determined. For example, when the object label of the corresponding first image is highly concentrated in a certain object label, it can be considered that the object label of the second image cluster is also the above-mentioned highly concentrated object label.

[0132] S105 : Determine, from the target video, target video segments corresponding to the respective object tags according to the object tags corresponding to the respective first image clusters and the second image clusters whose object tags have been determined.

[0133] Each object tag corresponds to at least one first image cluster and may also correspond to at least one second image cluster. These first and second image clusters are referred to as target image clusters. Each image in the target image cluster corresponds to a video frame, so the video frame can be placed in the video frame set of the object tag. All video frames in the video frame set are the target video segment. It should be understood that there are no duplicate video frames in a target video segment. For example, when a first image and the corresponding second image both correspond to the same video frame, since the first image and the second image are from the same video frame, the video frame will only be placed in the same target video segment once.

[0134] In some embodiments, the present application can set a unique identifier for each video frame in the target video, and when the first image and / or the second image in any video frame is detected, a unique identifier for the first image and / or the second image will be set, and a correspondence between the unique identifier of the first image and the unique identifier of the video frame, and a correspondence between the unique identifier of the second image and the unique identifier of the video frame will be established, laying the foundation for retrieving the target video segment corresponding to the video tag.

[0135] In an embodiment of the present application, multimodal (first and second part) detection is performed on at least a portion of video frames in a target video to obtain a first image set, a second image set, and a reference video frame set. Intra-modal association is performed on the first and second image sets to obtain at least one first image cluster and at least one second image cluster, and an object label is assigned to each first image cluster. The reference video frame set is used to determine the correspondence between first and second images belonging to the same target object. For each second image cluster, the object label of the second image cluster is determined based on the object label of the first image corresponding to the matching image in the second image cluster, thereby achieving inter-modal association. Based on the object labels corresponding to each first image cluster and the second image cluster with the determined object label, the target video segment corresponding to each object label is determined from the target video. In this application, intra-modal and inter-modal association is performed on features of multiple modalities. Based on the high accuracy of first part detection, the associated second image clusters can also correspond to the object label to a certain extent. When the first part is not visible, the second part provides additional guidance information, thereby improving the accuracy and recall of obtaining task highlights.

[0136] Based on the above embodiments, as an optional embodiment, the method further includes:

[0137] The audio corresponding to the target video is divided into multiple audio segments based on characters, and the multiple audio segments are clustered to obtain at least one audio segment cluster;

[0138] For each audio segment cluster, a matching degree between the audio segment cluster and each to-be-matched image cluster is determined, where the to-be-matched image cluster includes at least one of the first image cluster and the second image cluster.

[0139] The audio corresponding to the target video is divided into multiple audio clips based on the character as the unit, that is, the audio is detected in time sequence, and the audio is segmented whenever a change in the sound-generating object is detected. The embodiment of the present application can be based on the open source speech analysis toolkit pyannote-audio, and the complete audio content is divided into audio clips of different characters speaking (that is, whenever the speaking character changes, an audio clip is segmented), and the background noise is filtered out. At this time, it is only known that the speaking characters of any adjacent audio clips are different, but it is not known which character corresponds to which or which audio clips. By clustering the various audio clips, at least one audio clip cluster is obtained, and each audio clip cluster can be considered to correspond to a speaking character. In some embodiments, audio clips in which multiple people are speaking at the same time are discarded to avoid interference with subsequent speech clustering.

[0140] For the multiple audio clips obtained, embodiments of the present application can further obtain speech features for each audio clip, and use the speech features to cluster the multiple audio clips to obtain at least one audio clip cluster. It can be understood that the speech features of all audio clips in an audio clip cluster have a high degree of similarity. Embodiments of the present application do not specifically limit the specific method for extracting speech features. In some embodiments, the speech features of each audio clip can be extracted based on a thin-ResNet-34 model pre-trained on the speech dataset VoxCeleb2.

[0141] In variety shows, it's common to see a guest speaking when the camera suddenly cuts to another member of the audience. Using only visual features of a person's face or body, it's nearly impossible to recall the exact moment the target person was speaking. Voice recognition can address this pain point.

[0142] For each second image cluster, determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster includes:

[0143] For each second image cluster, the object label of the second image cluster is determined according to the object label of the first image corresponding to the matching image in the second image cluster and the matching degree between the second image cluster and each audio segment cluster.

[0144] When determining the object label of the second image cluster, the embodiment of the present application not only refers to the object label of the first image, but also considers the matching degree between the second image cluster and each audio segment cluster.

[0145] Since the video frame containing each image in the second image cluster has time information in the target video, and each audio segment in the audio segment cluster also has time information in the audio, the degree of overlap (i.e., time-series IOU) of the time information of each image in the second image cluster and the time information of each audio segment in each audio segment cluster can be calculated as the degree of matching between the second image cluster and each audio segment cluster. When the degree of matching between a second image cluster and an audio segment cluster is high, the object corresponding to the second image in the second image cluster, that is, the sound object of the audio segment, can be determined. When the degree of matching between the audio segment and a first image cluster is high at the same time, the association relationship between the second image cluster and the first image cluster is established. The embodiment of the present application uses the clustering results of three modes to determine the object label of the second image cluster, which specifically solves the problem of inaccurate recall of video frames when the shot content and audio are inconsistent.

[0146] Based on the above embodiments, as an optional embodiment, detecting the first part and the second part of the target object is performed on some video frames in the target video, which also includes:

[0147] The target video is divided into a plurality of video segments based on shots, and a preset number of video frames are determined from each video segment as at least a portion of the video frames.

[0148] Generally speaking, a shot is a continuous, visually uninterrupted segment captured by a camera. Shots can be categorized as short or long, typically ranging from a few seconds to several minutes in duration. With the exception of rare cases like moving shots, the subject often remains unchanged within a shot (for moving shots, this can be ensured by further segmenting the shot). Therefore, certain clear frames within a shot already contain information about the subject.

[0149] Based on this, the embodiments of the present application adopt a frame extraction method based on shot segmentation. In some embodiments, a pre-trained shot segmentation model is first used to cut the target video into several shots, and then a clear frame image is sampled from each shot to represent the shot. Specifically, after obtaining the video clips of each shot, the embodiments of the present application can input the video clips into a pre-trained clarity recognition model for each video clip, obtain the clarity of each frame image in the video clip output by the clarity recognition model, and use the image with the highest clarity to represent the shot. In some embodiments, the frame images in the video clip can also be input into the clarity recognition model in sequence, and when the first frame image whose clarity exceeds the preset clarity threshold appears, the frame image represents the shot. That is, the embodiments of the present application can use the first clear internal frame in each shot to represent the shot.

[0150] Specifically, the clarity recognition model can be trained as follows: First, a certain number of sample images are collected and the clarity of each sample image is obtained. Then, an initial model is trained based on the sample images and their clarity. The sample images are used as training samples, and the clarity of the sample images is used as sample labels to obtain the clarity recognition model. The initial model can be a single neural network model or a combination of multiple neural network models.

[0151] In some optional embodiments, the embodiments of the present application may use I-frames in video clips to represent shots. I-frames are also called intra pictures. I-frames are usually the first frame of each GOP. After appropriate compression, they can be used as a reference point for random access and can be treated as images. In the field of video transmission, when a video transmitter sends a video to a receiver, it does not always send complete pictures to the remote end. Instead, it only sends the part of the subsequent picture that has changed based on the previous picture. The I-frame is the reference point for all pictures. The embodiments of the present application directly use I-frames to represent shots, which can omit the step of detecting the clarity of the video frames in each video clip, significantly improving video processing efficiency.

[0152] Compared with the traditional frame-by-frame or alternate-frame extraction method, the frame extraction method based on shot segmentation in the embodiment of the present application utilizes the visual features of each shot to intelligently compress the amount of data to be processed, reduce feature redundancy, and more effectively retain the information of the characters appearing in the shot, thereby greatly improving the speed performance of the downstream process.

[0153] Based on the above embodiments, as an optional embodiment, determining the target video segments corresponding to the respective object tags from the target video includes:

[0154] For each object label, the first image cluster and the second image cluster with the object label are both used as target image clusters, and the video segment corresponding to each image in the target image cluster is used as the target video segment corresponding to the object label.

[0155] It should be understood that in the embodiment of the present application, each first image cluster has a corresponding object label. When there is at least one second image cluster with the same object label as the first image cluster, the target image cluster of the object label includes both the first image cluster and the second image cluster. When there is no second image cluster with the same object label as the first image cluster, the target image cluster of the object label only includes the first image cluster. Since each image in the target image cluster is extracted from a video frame selected from a video clip of a shot, when the image has an object label, the video frame where the image is located also has the object label, and further, the video clip where the video frame is located also has the object label. The present application divides the target video into multiple video clips based on the shot as a unit, determines a preset number of video frames from each video clip, and utilizes the prior knowledge that the objects in the shot are generally fixed. When an image corresponding to one or more object labels is determined in a video frame, it is considered that the video clip of the entire shot also corresponds to one or more object labels, thereby improving the efficiency of obtaining a collection of video frames of the object.

[0156] Based on the above embodiments, as an optional embodiment, determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster includes:

[0157] If the number of matching images in the second image cluster and the discreteness of the object labels of the first images corresponding to the matching images meet the first preset condition, the object label with the largest number among the object labels of all matching images is used as the object label of the second image cluster.

[0158] In the embodiments of the present application, when determining the object label of the second image cluster based on the object label of the first image cluster, two pieces of information are considered: one is the number of matching images. The greater the number of matching images, the more second images in the second image cluster have a clear correspondence between the first part and the second part, which means that the second image cluster has a clear object label (these second images with a corresponding relationship have a high probability of having the same object label as the corresponding first image). The other is the degree of discreteness of the object labels of the first images corresponding to the matching images. The higher the discreteness, the more mixed objects there are in the second image cluster, and the lower the probability that the second image cluster corresponds to the same object label. In some embodiments, when the proportion of the number of matching images in the second image cluster to the total number of images in the second image cluster exceeds a first value, and the discreteness is less than a second value, it is determined that the second images of the second image cluster are relatively uniform and all correspond to the same object label. The object label with the largest number of matching images among all the matching images is used as the object label of the second image cluster.

[0159] The present embodiment determines the number of matching images in the second image cluster and the degree of dispersion of the object labels of the corresponding first images. A greater number of matching images indicates a greater number of second images with clear object labels in the second image cluster, and a smaller degree of dispersion indicates a greater probability that the second image cluster corresponds to the same object label and a greater reliability of the object labels corresponding to each second image in the second image cluster. This application provides a feasible solution for determining the object labels of the second image cluster based on the correspondence relationship.

[0160] Based on the above embodiments, as an optional embodiment, determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster and the matching degree between each audio clip cluster and each to-be-matched image cluster includes:

[0161] If at least one of the number of matching images in the second image cluster or the degree of discreteness of the object labels of the first images corresponding to the matching images does not meet the first preset condition, the object label of the second image cluster is determined according to the degree of matching between each audio clip cluster and each image cluster to be matched.

[0162] The fusion process of the three modalities in the embodiment of the present application adopts a step-by-step fusion approach. First, the first image cluster is used to determine the most accurate image of each object, and then the correspondence between the first image and the second image is used to determine the second image cluster corresponding to each object. Although the accuracy of this step cannot be equivalent to using the first image cluster to determine the image of each object, the accuracy is still relatively high and feasible, and it can mine video frames when the object is unclear or does not appear in the first part. For the second image cluster where the corresponding object cannot be determined using the correspondence, it is necessary to further use the audio cluster for judgment.

[0163] It should be noted that when at least one of the number of matching images in the second image cluster or the discrete degree of the object label of the first image corresponding to the matching image does not meet the first preset condition, it means that the reliability of the object label of each image in the second image cluster is relatively low. At this time, the object label of the second image cluster is determined by using the matching degree between the audio clip cluster and each image cluster to be matched, which embodies the idea of determining the character label corresponding to the second image cluster step by step, and can effectively improve the recognition efficiency.

[0164] Based on the above embodiments, as an optional embodiment, determining the object label of the second image cluster according to the matching degree between each audio segment cluster and each to-be-matched image cluster includes:

[0165] If the matching degree between the second image cluster and a first image cluster and the same audio clip cluster meets a second preset condition, the object label corresponding to the first image cluster is used as the object label corresponding to the second image cluster.

[0166] In an embodiment of the present application, the degree of overlap between the time of the video frames corresponding to each image in the image cluster and the time of each audio segment in the audio segment cluster can be calculated. For example, if there are 3 images in the image cluster, the time of the video frames corresponding to the 3 images is 15:12, 16:30 and 17:58, and the time of the 3 audio segments in the audio segment cluster is [15:10, 15:14], [16:29, 16:31], and [17:58, 18:01]. The time of the 3 video frames is respectively within the time of the 3 audio segments. Therefore, it can be considered that each image has matching audio, and the degree of matching between the image cluster and the audio segment cluster is 100%.

[0167] When the degree of matching between the second image cluster and a first image cluster and the same audio clip cluster meets the second preset condition, it can be considered that the second image cluster has a high degree of temporal overlap with the first image cluster, and therefore the object label of the second image cluster can be set to the object label corresponding to the first image cluster.

[0168] Based on the above embodiments, as an optional embodiment, the object label of the second image cluster is determined according to the matching degree between each audio segment cluster and each to-be-matched image cluster, and the method also includes:

[0169] For each matching image in the second image cluster, a target object label of the first image corresponding to the matching image is determined, and the matching image is moved to a target second image cluster having the target object label.

[0170] In an embodiment of the present application, if the correspondence between the first image and the second image cannot be determined, the target object label of the first image corresponding to the matching image can be determined first, and then the matching image can be moved to the target second image cluster with the target object label.

[0171] In practical applications, the above situation often occurs when an object changes its appearance and the change of appearance lasts for a short time. If the correspondence between the first image and the second image has been determined, the target object label of the image cluster where the first image is located is first determined, so that the object corresponding to the first image can be clearly known. Then, the target second image cluster with the target object label is determined, that is, the feature set of the second part of the object is found, and the second image is moved to the target second image cluster, thereby realizing the recognition of the video frame when the object changes its appearance.

[0172] Based on the above embodiments, as an optional embodiment, determining the object label of the second image cluster according to the matching degree between each audio segment cluster and each to-be-matched image cluster includes:

[0173] If the degree of matching between the second image cluster and all audio clip clusters does not meet the second preset condition, then for each non-matching image in the second image cluster, the similarity between the non-matching image and the cluster center of each second image cluster is determined; if the difference between the maximum similarity and the second maximum similarity is greater than a preset threshold, the non-matching image is moved to the second image cluster with the maximum similarity.

[0174] In an embodiment of the present application, if a second image cluster has neither an object label determined based on the correspondence nor an object label determined based on the degree of matching with the audio segment cluster, then the non-matching image in the second image cluster can be tasked as an abnormal image. For abnormal images, the present application calculates their similarity with the cluster center of each second image cluster, where the cluster center is represented by the central feature of the second image cluster. If the difference between the maximum similarity and the second largest similarity is greater than a preset threshold, the non-matching image is moved to the second image cluster with the maximum similarity. This method is based on the assumption that when a sample belongs to a cluster, it should be close to the center of the cluster to which it belongs, and far enough away from the centers of other clusters. Through the comprehensive judgment of the three modalities, the human body labels of abnormal images can be assigned more accurately.

[0175] See Figure 3 , which exemplarily shows a flow chart of object label recognition based on multimodal features in an embodiment of the present application, as shown in the figure, including:

[0176] Clustering the images in the first image set to obtain at least one first image cluster. Since each image in the first image set is determined based on the detection of the first part, and the accuracy of the detection of the first part is quite high, it is believed that the accuracy of the classification result of the first image set is also quite high, and an object label can be set for each first image cluster.

[0177] Clustering the images of the second image set to obtain at least one second image cluster. Since each image in the second image cluster is determined based on detection of the second part, and the accuracy of detection of the second part is lower than the accuracy of detection of the first part, and since some subjects often change clothes in videos, it cannot be directly assumed that different second image clusters correspond to different subjects.

[0178] For each reference video frame, determining a correspondence between a first image and a second image belonging to the same target object;

[0179] determining whether the object labels of the first images corresponding to the matching images in the second image cluster are reliable based on the number of matching images in the second image cluster and the degree of dispersion of the object labels of the first images corresponding to the matching images;

[0180] If it is reliable, the object label with the largest number among all the object labels of the matching images is used as the object label of the second image cluster;

[0181] If unreliable, determining the target object label of the first image corresponding to the matching image, and moving the matching image to the target second image cluster having the target object label;

[0182] Determining a temporal matching degree between each first image cluster and each audio clip cluster; if the matching degree between the second image cluster and one first image cluster and the same audio clip cluster meets a second preset condition, using the object label corresponding to one first image cluster as the object label corresponding to the second image cluster;

[0183] If the degree of matching between the second image cluster and all audio clip clusters does not meet the second preset condition, then for each non-matching image in the second image cluster, determine the similarity between the non-matching image and the cluster center of each second image cluster; if the difference between the maximum similarity and the second maximum similarity is greater than a preset threshold, move the non-matching image to the second image cluster with the maximum similarity;

[0184] The non-matching image is a second image in the second image cluster that has no corresponding relationship.

[0185] Based on the above embodiments, as an optional embodiment, the target video is divided into multiple video segments based on shot units, including:

[0186] S201, predicting the segmentation confidence of each video frame using a pre-trained shot segmentation model;

[0187] S202: Counting the number of video frames whose segmentation confidences are higher than a segmentation threshold within a plurality of video frames using a sliding window. If the number exceeds a preset value, use the video frame with the highest segmentation confidence in the sliding window as a video segmentation point.

[0188] S203: Segment the target video according to the video segmentation points.

[0189] Many shot segmentation models have been disclosed in the relevant technology. By inputting a video into the shot segmentation model, the shot segmentation model can output the segmentation confidence of each video frame in the video. The higher the segmentation confidence, the greater the probability of using the video frame as a video segmentation point. Generally, a video frame with a segmentation confidence higher than a threshold can be used as a video segmentation point. During segmentation, whenever a video segmentation point is encountered, the video segmentation point is used as the first frame or the last frame of the video segment based on preset rules.

[0190] In some embodiments, the present application may use the shot segmentation model TransNetv2 to predict the segmentation confidence of each video frame.

[0191] It's important to note that typical shot segmentation models use a fixed segmentation threshold, where any frame above the threshold is considered a segmentation point. However, in many video scenarios, such as broadcasts and variety shows, moving shots and special transitions are common, and the frames within these shots often have a high segmentation confidence. Using a fixed segmentation threshold can result in overly fragmented shots, even leading to the loss of important frames.

[0192] To address this issue, the present application optimizes a fixed segmentation threshold, using a sliding window to count the number of frames within a video segment that exceed the threshold. If this number is high, it is determined to be a moving shot or a special transition shot, and only the frames with the highest confidence level within the sliding window are selected as segmentation points. This method of dynamically selecting video segmentation points based on a sliding window has been verified to significantly improve the robustness of shot segmentation.

[0193] In broadcasting or variety shows, the same subject usually wears different clothes when attending different events or participating in different programs, making it difficult to recall the sample after changing clothes based solely on appearance characteristics. Figure 4This example shows the first and second images of a subject who changed multiple outfits during a show (represented by different patterns on the clothing). For simplicity, this example assumes that the subject maintains the same pose during sampling. In some shots, the subject's face is facing away or sideways from the camera, resulting in no valid face recognition. "X" in the figure indicates a non-recognized face. Recalling samples after clothing changes based solely on body and clothing features is extremely challenging.

[0194] To address this issue, the present invention provides a multimodal neighbor feature optimization method, which includes three parts: intra-modal correlation, inter-modal correlation, and second feature (i.e., feature of the second part) update. Specifically, Figure 5 Shown, including:

[0195] S301: Obtain a first feature of each first image and an initial second feature of each second image.

[0196] It should be understood that in this application, the first feature of the first image is the feature of the first part in the first image, and the initial second feature of the second image is the initial feature of the second part in the second image. Taking the second image as an example, if the total number of second images is N, then the set of initial second features of all second images can be expressed as F b =[f1, f2, ..., f N ].

[0197] S302. Determine each nearest neighboring first image pair based on the first feature of each first image, and determine each nearest neighboring second image pair based on the initial second feature of each second image, wherein the two first images in each nearest neighboring first image pair are nearest neighbors to each other; and the two second images in each nearest neighboring second image pair are nearest neighbors to each other.

[0198] When calculating the nearest neighbor image of each image, the present application may use Euclidean distance for measurement. Of course, other distances, such as Mahalanobis distance, cosine distance, etc., may also be used. The embodiments of the present application do not make specific limitations.

[0199] Taking the set of initial second features of the second image as an example, the matrix consisting of the distances between any two second images can be obtained:

[0200]

[0201] Taking d12 as an example, d12 represents the distance between the initial second feature of the second image 1 and the initial second feature of the second image 2.

[0202] Matrix D N×NEach row in represents a distance vector for a second image, that is, the distance between a second image and all other second images. By sorting the N values in the distance vector by size, we can obtain the K nearest neighbor images (Top-K neighbors) of each second image.

[0203] If two second images are each other's nearest neighbors, then the two second images may constitute a nearest neighbor second image pair. For example, if second image i's K nearest neighbors include second image j, and second image j's K nearest neighbors include second image i, then second image i and second image j constitute a nearest neighbor second image pair.

[0204] Similarly, each nearest first image pair can be determined by the above method.

[0205] S303 : Update the nearest second image pair containing the second images corresponding to the two first images according to the correspondence between the two first images in each nearest first image pair.

[0206] In some embodiments, for each nearest neighbor first image pair, the two first images in the nearest neighbor first image pair are determined; if both first images have corresponding target second images, the two corresponding target second images are determined as the nearest second image pair. Specifically, if first image a and first image b form a nearest neighbor first image pair, and first image a corresponds to second image i, and first image b corresponds to second image j, then second image i and second image j are also considered to form a nearest second image pair. If second image i and second image j are not considered as the nearest first image pair in step S402, a nearest second image pair consisting of second image i and second image j needs to be created.

[0207] S304 . For each second image, use the nearest image in all updated nearest second image pairs of the second image as the target nearest image, and obtain the target second feature of the second image according to the initial second feature of the target nearest image.

[0208] After the embodiment of the present application updates the nearest second image pair of the second image, the initial second feature of the second image can be updated. Specifically, it can be calculated according to the following formula:

[0209]

[0210] Among them, f i represents the i-th second image, α∈(0,1), represents the weight coefficient, β m represents the weight of the nearest neighbor image in the mth nearest neighbor image pair, which is inversely proportional to the distance and satisfies β1+β2+…+β M=1, M represents the total number of the nearest second image pairs of the i-th second image.

[0211] S305 : Clustering the first features of all first images to obtain at least one first image cluster; and clustering the target second features of all second images to obtain at least one second image cluster.

[0212] See Figure 6 , which exemplarily shows a flow chart of a video processing method of another embodiment of the present application. As shown in the figure, the method generally includes five steps, namely, frame extraction based on shot segmentation, multimodal detection, feature extraction (Embedding), multimodal feature optimization, and multimodal fusion.

[0213] Specifically, for the frame extraction steps based on shot segmentation:

[0214] In the embodiment of the present application, a target video is divided into a plurality of video segments based on shots, and a preset number of video frames are determined from each video segment as at least a portion of the video frames;

[0215] Steps for multimodal detection:

[0216] The embodiment of the present application detects a first part and a second part of a target object in the extracted video frame to obtain a first image set and a second image set; for the audio corresponding to the target video, the embodiment of the present application divides the audio into multiple audio segments based on the person as a unit to obtain an audio segment set;

[0217] For feature extraction steps:

[0218] In the embodiment of the present application, feature extraction is performed on the images in the first image set to obtain a first feature set, and feature extraction is performed on the images in the second image set to obtain a second feature set; voiceprint recognition is performed on each audio clip to obtain a voiceprint feature set;

[0219] Steps for multimodal feature optimization:

[0220] The present application determines the correspondence between a first image and a second image belonging to the same target object by referring to a video frame, and performs intra-modal association on the first feature set and the second feature set, that is, determining each nearest first image pair and each nearest second image pair based on the Top-K nearest neighbor algorithm; then updating the nearest second image pair based on each nearest first image pair and the correspondence; and using the updated nearest second image pair to update the second feature of the second image;

[0221] Steps for multimodal fusion:

[0222] In this embodiment of the present application, images in a first image set are clustered to obtain at least one first image cluster, images in a second image set are clustered to obtain at least one second image cluster, and multiple audio clips are clustered to obtain at least one audio clip cluster; an object label is set for each first image cluster;

[0223] If the number of matching images in the second image cluster and the discreteness of the object labels of the first image corresponding to the matching images meet a first preset condition, the object label with the largest number among the object labels of all matching images is used as the object label of the second image cluster;

[0224] If at least one of the number of matching images in the second image cluster or the discreteness of the object label of the first image corresponding to the matching image does not meet the first preset condition, determining the target object label of the first image corresponding to the matching image, and moving the matching image to the target second image cluster having the target object label;

[0225] If the matching degree between the second image cluster and a first image cluster and the same audio clip cluster meets a second preset condition, the object label corresponding to the first image cluster is used as the object label corresponding to the second image cluster;

[0226] If the degree of matching between the second image cluster and all audio clip clusters does not meet the second preset condition, then for each non-matching image in the second image cluster, the similarity between the non-matching image and the cluster center of each second image cluster is determined; if the difference between the maximum similarity and the second maximum similarity is greater than a preset threshold, the non-matching image is moved to the second image cluster with the maximum similarity.

[0227] The application scenarios of this application include character material retrieval, character "only viewing" in video playback, video content recommendation, etc.

[0228] Character material retrieval refers to the efficient retrieval of characters based on their unique identifiers. It can locate characters at different granularities (video, time interval, frame), and support secondary production of character dimensions, such as Figure 7 As shown, the video is processed by the video processing method provided in the embodiment of the present application to obtain video clips of multiple objects, that is, character highlights, and each character highlight is stored in the media library. When the detection party wants to retrieve the character highlights of a certain object, it sends a retrieval request to the media library, for example, to obtain the character highlights of Wang Wu. The media library obtains the character highlights of Wang Wu according to the retrieval request and returns it to the search party.

[0229] "Only viewing" of characters in video playback means that during video playback, you can quickly jump to or filter content by character tags. Figure 8As shown. When the viewer decides to only watch the character "Zhang San" in the variety show, only the video frames in which "Zhang San" appears are displayed on the video playback timeline, and the video frames in which "Zhang San" does not appear are grayscaled or hidden. The user will not be able to select the video frames in which "Zhang San" does not appear. Therefore, when the user slides the timeline, only the video frames in which "Zhang San" appears will be jumped to and played.

[0230] See Figure 9 , which exemplarily shows a flow chart of a video processing method according to another embodiment of the present application, as shown in the figure, including:

[0231] S401. Detect a first part and a second part of a target object on at least part of the video frames in a target video to obtain a first image set and a second image set; each first image in the first image set is a subregion of the first part of a target object in a video frame; each second image in the second image set is a subregion of the second part of a target object in a video frame.

[0232] The method of detecting the first part and the second part of the video frame in the target video in the embodiment of the present application can refer to the above embodiments and will not be repeated here.

[0233] S402: Divide the audio corresponding to the target video into multiple audio segments based on characters, cluster the multiple audio segments to obtain at least one audio segment cluster; cluster the images in the first image set to obtain at least one first image cluster, cluster the images in the second image set to obtain at least one second image cluster, and set an object label for each first image cluster.

[0234] The manner in which the audio clips, the first image set, and the second image set are clustered in the embodiment of the present application can be referred to in the above embodiments and will not be described in detail here.

[0235] S403 : For each audio segment cluster, determine the matching degree between the audio segment cluster and each to-be-matched image cluster, where the to-be-matched image cluster includes at least one of the first image cluster and the second image cluster.

[0236] In the embodiment of the present application, the method of determining the matching degree between the audio segment cluster and each to-be-matched image cluster can refer to the above embodiments and will not be described in detail here.

[0237] S404. For any first image cluster, an audio segment cluster whose matching degree with the first image cluster meets a second preset condition is used as a reference audio cluster, and a second image cluster whose matching degree with the reference audio cluster meets the second preset condition is used as a reference second image cluster. The target video frame corresponding to the reference audio cluster and the object label of the reference second image cluster are determined as the object label of the first image cluster.

[0238] For the methods of determining the degree of matching between the first image cluster and the audio segment cluster and the degree of matching between the second image cluster and the audio segment cluster, the embodiments of the present application can refer to the above-mentioned embodiments and will not be repeated here. For any first image cluster, if the degree of matching between the first image cluster and an audio segment cluster (reference audio cluster) meets the second preset condition, it can be determined that the speaker of the audio segment cluster is the same object as the object in the first image cluster. Since each audio segment in the reference audio cluster has a corresponding video frame, that is, a target video frame, the object label of the target video frame can be set as the object label of the first image cluster. In some videos, there may be a situation where the object shown in the video is inconsistent with the speaker of the audio. In response to such situations, the embodiments of the present application can also include the video frame in which the object is speaking in the video collection of the object, thereby enriching the video frames in the video collection.

[0239] In addition, this application uses the audio clip cluster as a link to establish the association between the first image and the second image. If a first image cluster and a second image cluster both have a high degree of matching with an audio clip cluster, it means that the object corresponding to the second image cluster is the same object as the objects of the first image cluster and the audio clip cluster. Therefore, the object label of the second image cluster is also the object label of the first image cluster.

[0240] S405 : Determine target video segments corresponding to the respective object tags from the target video according to the respective first image clusters, the second image clusters with determined object tags, and the object tags corresponding to the target video frames.

[0241] In an embodiment of the present application, multimodal (first part, second part, audio) detection is performed on at least a portion of video frames in a target video to obtain a first image set, a second image set, and an audio clip set. Intra-modal association is then performed on the first image set, the second image set, and the audio clip set to obtain at least one first image cluster, at least one second image cluster, and at least one audio clip cluster. An object label is assigned to each first image cluster. When the degree of match between the audio clip cluster and the first image cluster meets a second preset condition, the audio clip cluster and the first image cluster are considered to correspond to the same object. When the degree of match between the audio clip cluster and both a first image cluster and a second image cluster meets the second preset condition, the audio clip cluster, the first image cluster, and the second image cluster are considered to correspond to the same object. Object labels for the audio clip cluster and the second image cluster are determined in this manner, achieving inter-modal association. In this manner, the present embodiment can expand the video collection to include video frames where only the sound of the object is heard but not seen. This means that video frames containing only the sound of the object are also included in the video collection of the object, enriching the content of the video collection.

[0242] Based on the above embodiments, as an optional embodiment, detecting the first part and the second part of the target object on at least part of the video frames in the target video further includes:

[0243] A reference video frame set is obtained, where each reference video frame in the reference video frame set contains at least one first image and at least one second image.

[0244] The method of obtaining the reference video frame set in the embodiment of the present application can refer to the above embodiments and will not be repeated here.

[0245] The method of the embodiment of the present application also includes:

[0246] For each reference video frame, determining a correspondence between a first image and a second image belonging to the same target object;

[0247] For each second image cluster, the object label of the second image cluster is determined according to the object label of the first image corresponding to the matching image in the second image cluster; the matching image is the second image in the second image cluster with a corresponding relationship.

[0248] The embodiment of the present application determines the correspondence between the first image and the second image belonging to the same target object by referring to the video frame, and then uses the correspondence to associate the second image cluster with the first image cluster and determine the object label of the second image cluster.

[0249] Based on the above embodiments, as an optional embodiment, detecting the first part and the second part of the target object is performed on some video frames in the target video, which also includes:

[0250] Dividing the target video into a plurality of video segments based on shot units, and determining a preset number of video frames from each video segment as at least a portion of the video frames;

[0251] Based on the above embodiments, as an optional embodiment, determining the target video segments corresponding to the respective object tags from the target video includes:

[0252] For each object label, the first image cluster and the second image cluster with the object label are both used as target image clusters, and the video segment corresponding to each image in the target image cluster is used as the target video segment corresponding to the object label.

[0253] Determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster includes:

[0254] If the number of matching images in the second image cluster and the discreteness of the object labels of the first images corresponding to the matching images meet the first preset condition, the object label with the largest number among the object labels of all matching images is used as the object label of the second image cluster.

[0255] Based on the above embodiments, as an optional embodiment, determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster and the matching degree between each audio clip cluster and each to-be-matched image cluster includes:

[0256] If at least one of the number of matching images in the second image cluster or the degree of discreteness of the object labels of the first images corresponding to the matching images does not meet the first preset condition, the object label of the second image cluster is determined according to the degree of matching between each audio clip cluster and each image cluster to be matched.

[0257] Based on the above embodiments, as an optional embodiment, determining the object label of the second image cluster according to the matching degree between each audio segment cluster and each to-be-matched image cluster includes:

[0258] If the matching degree between the second image cluster and a first image cluster and the same audio segment cluster meets the second preset condition, the object label corresponding to the first image cluster is used as the object label corresponding to the second image cluster.

[0259] Based on the above embodiments, as an optional embodiment, the object label of the second image cluster is determined according to the matching degree between each audio segment cluster and each to-be-matched image cluster, and the method also includes:

[0260] For each matching image in the second image cluster, a target object label of the first image corresponding to the matching image is determined, and the matching image is moved to a target second image cluster having the target object label.

[0261] Based on the above embodiments, as an optional embodiment, determining the object label of the second image cluster according to the matching degree between each audio segment cluster and each to-be-matched image cluster includes:

[0262] If the degree of matching between the second image cluster and all audio clip clusters does not meet the second preset condition, then for each non-matching image in the second image cluster, determine the similarity between the non-matching image and the cluster center of each second image cluster; if the difference between the maximum similarity and the second maximum similarity is greater than a preset threshold, move the non-matching image to the second image cluster with the maximum similarity;

[0263] The non-matching image is a second image in the second image cluster that has no corresponding relationship.

[0264] The following describes the video processing method of this application in conjunction with a specific application scenario. Figure 10 , which exemplarily shows a structural diagram of a video processing system of the present application, wherein the video processing system includes a user terminal, a video database, a feature detection server, and a video server.

[0265] The user terminal can be connected to the video server through the network, and the user terminal can be the user terminal of any video viewer. The embodiment of the present application does not limit the type of video playback application running on the user terminal. It can be a video playback application that the user needs to download and install, or it can be a cloud video playback application, or it can be a video playback application in a small program. When the video playback application is running, the user terminal sends a video playback request to the video server through the network. The video server obtains the corresponding target video and related information of the target video from the video database based on the video playback request, and returns it to the user terminal. The related information of the target video may include information about each character in the target video.

[0266] The user terminal can play the target video, and in response to the video viewer selecting at least one target role from the protagonist information, a video highlights acquisition request is generated, wherein the video highlights acquisition request includes the target role information.

[0267] The video server sends the target video to the feature detection server, which includes a pre-trained shot segmentation model, a first part detection model, a second part detection model, and a voiceprint detection model. The shot segmentation model is used to divide the target video into shot units to obtain multiple video clips. Each video clip is sent to the first part detection model and the second part detection model respectively to obtain a first image set and a second image set. A reference video frame set can be obtained based on the video frames in which each image in the first image set and the second image set is located. The voiceprint detection model divides the audio corresponding to the target video into multiple audio clips based on characters. The first part detection model, the second part detection model, and the voiceprint detection model can also respectively extract features from the first image, the second image, and the audio clip to obtain corresponding first features, second features, and voiceprint features.

[0268] The video server clusters the images in the first image set to obtain at least one first image cluster, clusters the images in the second image set to obtain at least one second image cluster, clusters the voiceprint features to obtain at least one audio segment cluster, sets an object label for each of the first image clusters, and determines the object label of each second image cluster based on the object label of the first image corresponding to the matching image in the second image cluster and the degree of matching between the second image cluster and each audio segment cluster.

[0269] The video server uses the object tag corresponding to the target character as the target object tag, the first image cluster and the second image cluster with the target object tag as the target image cluster, and the video clip corresponding to each image in the target image cluster as the target video clip corresponding to the object tag, and generates a character highlight based on the target video clip and returns it to the user terminal.

[0270] From this we can see that in this character collection task, the input data is a complete video, and the output result is all the time periods in which the character to be found or the main character appears effectively in this video.

[0271] If a complete video S is a sequence of n frames [f1, f2, ...f n ], frame i f i Represents a time point t on the time axis i , so this video can also be represented as a time point sequence [t1, t2, ...t n ]. Let m main characters to be searched be represented as p1, p2, ... p m After being processed by the embodiment of the present application, all the time points at which each main character effectively appears can be output to form a subsequence [t1, t2, ...t i ]. For example, for the jth main character, the process can be expressed as:

[0272] G(S, p j )=[t1,t2,...t i ]

[0273] To further simplify the representation, if the main character appears in every frame within a time period, all time points within this time period can be represented as a closed interval [t s , t e ], where t s and t e Indicates the time when the main character first appears and the time when he last appears in this time period. The above formula can be converted into:

[0274] G(S, p j )=[[t1,t i ]]

[0275] This embodiment of the application uses a shot-segmented frame extraction method. Compared to traditional frame-by-frame or alternate-frame extraction methods, this method significantly reduces the total number of frames extracted and more effectively preserves the information of the characters appearing in the shot. As a result, the amount of downstream processing data is significantly reduced, significantly improving the overall efficiency of the algorithm.

[0276] This application also conducted a test experiment to demonstrate the improvement in algorithm efficiency achieved by using a shot-segmentation-based frame extraction method. Using a 90-minute variety show video with 137,014 frames as the test video, the shot-segmentation-based frame extraction method extracted 1,475 frames. The alternate frame extraction method, by contrast, extracted one frame every 1 second, for a total of 5,229 frames.

[0277] Aside from the frame extraction method, the two solutions share the same subsequent workflow. Table 1 shows the time taken for each algorithm component of the two solutions (the components involving deep learning model inference were run on a Tesla V100 graphics card):

[0278]

[0279]

[0280] Table 1 Comparison of time consumption of different frame extraction methods

[0281] As can be seen from Table 1, the frame extraction method based on shot segmentation in the embodiment of the present application saves nearly two-thirds of the processing time compared to the alternate frame extraction method, greatly improving the processing efficiency of the algorithm.

[0282] Furthermore, this application adopts a clustering fusion solution based on multimodal features, which can significantly improve the accuracy and recall rate of character highlights compared to related technologies that only cluster based on single-modal features of faces.

[0283] In order to illustrate the improvement of the algorithm performance of this solution, the inventor conducted a test experiment. 15 long variety show videos were used as test videos. The evaluation indicators were instance-level Pairwise F-score, Bcubed F-score, NMI and shot-level Shot Recall. The experimental results were the average evaluation results of the 15 videos. Solution 1 used the single-modal features of the face for clustering, while Solution 2 clustered the features of multiple modalities and then fused the multiple clustering results. The following table shows the algorithm performance of the two solutions (the part involving deep learning model inference was run on a Tesla V100 graphics card):

[0284]

[0285] Table 2 Performance comparison of different clustering methods

[0286] In Table 2, the Pairwise F-score measures classification accuracy, the Bcubed F-score comprehensively measures classification recall and precision, the Normalized Mutual Information (NMI) measures the similarity of clustering results, and the Shot Recall rate measures the percentage of all accurate character highlights retrieved. The results show that Solution 2 achieves higher precision and recall than Solution 1, with a particularly significant improvement of approximately 12 percentage points in Shot Recall. This result further demonstrates the importance of multimodal information for character highlights in complex scenes.

[0287] The embodiment of the present application provides a video processing device, such as Figure 11 As shown, the video processing device may include: a multi-mode detection module 101, a correspondence determination module 102, a clustering module 103, a label determination module 104 and a video highlights module 105, wherein:

[0288] The multi-mode detection module 101 is configured to detect a first portion and a second portion of a target object in at least a portion of video frames in a target video, thereby obtaining a first image set, a second image set, and a reference video frame set; each first image in the first image set is a subregion of the first portion of a target object in a video frame; each second image in the second image set is a subregion of the second portion of a target object in a video frame; and each reference video frame in the reference video frame set contains both at least one first image and at least one second image.

[0289] a correspondence determination module 102 for determining, for each reference video frame, a correspondence between a first image and a second image belonging to the same target object;

[0290] a clustering module 103 for clustering the images in the first image set to obtain at least one first image cluster, clustering the images in the second image set to obtain at least one second image cluster, and setting an object label for each first image cluster;

[0291] a label determination module 104 configured to determine, for each second image cluster, an object label of the second image cluster based on an object label of a first image corresponding to a matching image in the second image cluster; the matching image being a second image in the second image cluster having a corresponding relationship;

[0292] The video highlights module 105 is configured to determine target video segments corresponding to respective object labels from the target video according to the object labels corresponding to the respective first image clusters and the second image clusters with determined object labels.

[0293] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0294] Based on the above embodiments, as an optional embodiment, the multimode detection module is further configured to:

[0295] The audio corresponding to the target video is divided into multiple audio segments based on characters, and the multiple audio segments are clustered to obtain at least one audio segment cluster;

[0296] For each audio segment cluster, determining a matching degree between the audio segment cluster and each to-be-matched image cluster, where the to-be-matched image cluster includes at least one of the first image cluster and the second image cluster;

[0297] The tag determination module is specifically used to:

[0298] For each second image cluster, the object label of the second image cluster is determined according to the object label of the first image corresponding to the matching image in the second image cluster and the matching degree between the second image cluster and each audio segment cluster.

[0299] Based on the above embodiments, as an optional embodiment, the device further includes:

[0300] a shot segmentation module, configured to segment a target video into a plurality of video segments based on a shot, and determine a preset number of video frames from each video segment as at least a portion of the video frames;

[0301] The video highlights module is specifically used for:

[0302] For each object label, the first image cluster and the second image cluster with the object label are both used as target image clusters, and the video segment corresponding to each image in the target image cluster is used as the target video segment corresponding to the object label.

[0303] Based on the above embodiments, as an optional embodiment, the label determination module is specifically used to: if the number of matching images in the second image cluster and the discrete degree of the object labels of the first image corresponding to the matching images meet the first preset condition, then the object label with the largest number among the object labels of all matching images is used as the object label of the second image cluster.

[0304] Based on the above embodiments, as an optional embodiment, the tag determination module is specifically configured to:

[0305] If at least one of the number of matching images in the second image cluster or the degree of discreteness of the object labels of the first images corresponding to the matching images does not meet the first preset condition, the object label of the second image cluster is determined according to the degree of matching between each audio clip cluster and each image cluster to be matched.

[0306] Based on the above embodiments, as an optional embodiment, the tag determination module is specifically configured to:

[0307] If the matching degree between the second image cluster and a first image cluster and the same audio clip cluster meets a second preset condition, the object label corresponding to the first image cluster is used as the object label corresponding to the second image cluster.

[0308] Based on the above embodiments, as an optional embodiment, the device further includes:

[0309] The image moving module is used to determine, for each matching image in the second image cluster, a target object label of the first image corresponding to the matching image, and move the matching image to a target second image cluster having the target object label.

[0310] Based on the above embodiments, as an optional embodiment, based on the above embodiments, as an optional embodiment, the tag determination module is specifically configured to:

[0311] If the degree of matching between the second image cluster and all audio clip clusters does not meet the second preset condition, then for each non-matching image in the second image cluster, determine the similarity between the non-matching image and the cluster center of each second image cluster; if the difference between the maximum similarity and the second maximum similarity is greater than a preset threshold, move the non-matching image to the second image cluster with the maximum similarity;

[0312] The non-matching image is a second image in the second image cluster that has no corresponding relationship.

[0313] Based on the above embodiments, as an optional embodiment, the shot segmentation module includes:

[0314] Confidence unit, used to predict the segmentation confidence of each video frame through the pre-trained shot segmentation model;

[0315] A sliding window unit is used to count the number of video frames whose segmentation confidence is higher than a segmentation threshold within a plurality of video frames through a sliding window. If the number exceeds a preset value, the video frame with the highest segmentation confidence in the sliding window is used as the video segmentation point;

[0316] The segmentation module is used to segment the target video according to the video segmentation points.

[0317] Based on the above embodiments, as an optional embodiment, the clustering module includes:

[0318] an initial feature unit, configured to obtain a first feature of each first image and an initial second feature of each second image;

[0319] a neighboring image pair unit, configured to determine each nearest neighboring first image pair based on the first feature of each first image, and to determine each nearest neighboring second image pair based on the initial second feature of each second image, wherein the two first images in each nearest neighboring first image pair are nearest neighbors to each other, and the two second images in each nearest neighboring second image pair are nearest neighbors to each other;

[0320] an image pair updating unit, configured to update, based on the correspondence between the two first images in each of the most adjacent first image pairs, the most adjacent second image pairs where the second images corresponding to the two first images are located;

[0321] a feature updating unit configured to, for each second image, use a nearest image in all updated nearest-neighbor second image pairs of the second image as a target nearest-neighbor image, and obtain a target second feature of the second image based on an initial second feature of the target nearest-neighbor image;

[0322] The clustering unit is configured to cluster the first features of all first images to obtain at least one first image cluster; and cluster the target second features of all second images to obtain at least one second image cluster.

[0323] Based on the above embodiments, as an optional embodiment, the image pair updating unit is specifically configured to:

[0324] For each nearest first image pair, two first images in the nearest first image pair are determined; if both first images have corresponding target second images, the two corresponding target second images are determined as the nearest second image pair.

[0325] The embodiment of the present application provides a video processing device, such as Figure 12 As shown, the video processing device may include: a video frame part detection module 201, a multi-mode clustering module 202, an audio matching module 203, a multi-mode matching module 204 and a video segment screening module 205. Specifically:

[0326] The video frame part detection module 201 is configured to detect a first part and a second part of a target object in at least a portion of the video frames in the target video, thereby obtaining a first image set and a second image set; each first image in the first image set is a subregion of the first part of a target object in a video frame; and each second image in the second image set is a subregion of the second part of a target object in a video frame.

[0327] The multi-mode clustering module 202 is configured to segment the audio corresponding to the target video into multiple audio segments based on characters, cluster the multiple audio segments to obtain at least one audio segment cluster; cluster the images in the first image set to obtain at least one first image cluster, cluster the images in the second image set to obtain at least one second image cluster, and assign an object label to each of the first image clusters;

[0328] an audio matching module 203 configured to determine, for each audio segment cluster, a degree of matching between the audio segment cluster and each to-be-matched image cluster, wherein the to-be-matched image cluster includes at least one of the first image cluster or the second image cluster;

[0329] The multi-modal matching module 204 is configured to, for any first image cluster, use an audio segment cluster whose matching degree with the first image cluster meets a second preset condition as a reference audio cluster, use a second image cluster whose matching degree with the reference audio cluster meets the second preset condition as a reference second image cluster, and determine the target video frame corresponding to the reference audio cluster and the object label of the reference second image cluster as the object label of the first image cluster;

[0330] The video segment screening module 205 is configured to determine target video segments corresponding to respective object tags from the target video based on the respective first image clusters, the second image clusters with determined object tags, and the object tags corresponding to the target video frames.

[0331] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0332] Based on the above embodiments, as an optional embodiment, the video frame part detection module is further configured to:

[0333] Obtaining a reference video frame set, wherein each reference video frame in the reference video frame set simultaneously contains at least one first image and at least one second image;

[0334] The device also includes:

[0335] a correspondence module, configured to determine, for each reference video frame, a correspondence between a first image and a second image belonging to the same target object;

[0336] A second image cluster label determination module is configured to determine, for each second image cluster, an object label of the second image cluster based on an object label of a first image corresponding to a matching image in the second image cluster; the matching image is a second image in the second image cluster having the corresponding relationship.

[0337] In an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the video processing method. Compared with the related art, the following can be achieved: by performing multimodal (first part and second part) detection on at least part of the video frames in the target video, a first image set and a second image set and a reference video frame set are obtained, and the first image set and the second image set are respectively subjected to intra-modal association to obtain at least one first image cluster and at least one second image cluster, and an object label is set for each first image cluster; using the reference video frame set, the correspondence between the first image and the second image belonging to the same target object is determined. For each second image cluster, the object label of the second image cluster is determined based on the object label of the first image corresponding to the matching image in the second image cluster, achieving inter-modal association. Based on the object labels corresponding to each first image cluster and the second image cluster with determined object labels, the target video segment corresponding to each object label is determined from the target video. This application uses features from multiple modalities to perform intra-modal and inter-modal associations. Based on the high accuracy of first part detection, the associated second image clusters can also correspond to object labels to a certain extent. When the first part is not visible, the second part provides additional guidance, improving the accuracy and recall of obtaining task highlights.

[0338] In an alternative embodiment, an electronic device is provided, such as Figure 13 As shown, Figure 13 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0339] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0340] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0341] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0342] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0343] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0344] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0345] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0346] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0347] The above are only optional implementation methods for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A video processing method, characterized in that: include: Detecting a first part and a second part of a target object on at least a portion of video frames in a target video to obtain a first image set, a second image set, and a reference video frame set; each first image in the first image set is a subregion of the first part of a target object in a video frame; each second image in the second image set is a subregion of the second part of a target object in a video frame; each reference video frame in the reference video frame set simultaneously contains at least one first image and at least one second image; and the detection accuracy of the first part is high; For each of the reference video frames, determining a correspondence between a first image and a second image belonging to the same target object; Clustering the images in the first image set to obtain at least one first image cluster, clustering the images in the second image set to obtain at least one second image cluster, and setting an object label for each of the first image clusters; For each second image cluster, determining an object label of the second image cluster according to an object label of a first image corresponding to a matching image in the second image cluster; the matching image is a second image in the second image cluster having the corresponding relationship; According to the object labels corresponding to the first image clusters and the second image clusters whose object labels have been determined, target video segments corresponding to the respective object labels are determined from the target video.

2. The method according to claim 1, characterized in that Also includes: dividing the audio corresponding to the target video into a plurality of audio segments based on characters, and clustering the plurality of audio segments to obtain at least one audio segment cluster; For each audio segment cluster, determining a matching degree between the audio segment cluster and each to-be-matched image cluster, wherein the to-be-matched image cluster includes at least one of the first image cluster or the second image cluster; For each second image cluster, determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster includes: For each second image cluster, the object label of the second image cluster is determined according to the object label of the first image corresponding to the matching image in the second image cluster and the matching degree between the second image cluster and each audio segment cluster.

3. The method according to claim 1, characterized in that The detecting of the first part and the second part of the target object on some video frames in the target video further includes: Dividing the target video into a plurality of video segments based on shot units, and determining a preset number of video frames from each of the video segments as the at least partial video frames; The determining of target video segments corresponding to the respective object tags from the target video includes: For each object label, the first image cluster and the second image cluster having the object label are both used as target image clusters, and the video segment corresponding to each image in the target image cluster is used as the target video segment corresponding to the object label.

4. The method according to any one of claims 1 to 3, characterized in that The determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster includes: If the number of the matching images in the second image cluster and the discreteness of the object labels of the first images corresponding to the matching images meet a first preset condition, the object label with the largest number among the object labels of all the matching images is used as the object label of the second image cluster.

5. The method according to claim 4, characterized in that Determining the object label of the second image cluster according to the object label of the first image corresponding to the matching image in the second image cluster and the matching degree between each audio clip cluster and each to-be-matched image cluster includes: If at least one of the number of matching images in the second image cluster or the degree of discreteness of the object labels of the first images corresponding to the matching images does not meet the first preset condition, the object label of the second image cluster is determined based on the degree of matching between each of the audio clip clusters and each of the image clusters to be matched.

6. The method according to claim 5, characterized in that The determining the object label of the second image cluster according to the degree of matching between each of the audio clip clusters and each of the image clusters to be matched includes: If the matching degrees between the second image cluster and a first image cluster and the same audio clip cluster meet a second preset condition, the object label corresponding to the first image cluster is used as the object label corresponding to the second image cluster.

7. The method according to claim 5, characterized in that The determining of the object label of the second image cluster according to the degree of matching between each of the audio segment clusters and each of the image clusters to be matched may also include: For each matching image in the second image cluster, a target object label of the first image corresponding to the matching image is determined, and the matching image is moved to a target second image cluster having the target object label.

8. The method according to claim 6, characterized in that The determining the object label of the second image cluster according to the degree of matching between each of the audio clip clusters and each of the image clusters to be matched includes: If the degree of matching between the second image cluster and all audio clip clusters does not meet the second preset condition, then, for each non-matching image in the second image cluster, determining the similarity between the non-matching image and the cluster center of each second image cluster, and if the difference between the maximum similarity and the second maximum similarity is greater than a preset threshold, moving the non-matching image to the second image cluster with the maximum similarity; The non-matching image is a second image in the second image cluster that does not have the corresponding relationship.

9. The method according to claim 3, characterized in that The target video is divided into multiple video segments based on shots, including: Predict the segmentation confidence of each video frame through a pre-trained shot segmentation model; Counting the number of video frames whose segmentation confidence is higher than a segmentation threshold within a plurality of video frames through a sliding window, and if the number exceeds a preset value, taking the video frame with the highest segmentation confidence in the sliding window as the video segmentation point; The target video is segmented according to the video segmentation points.

10. The method according to claim 1, characterized in that Clustering the images in the first image set to obtain at least one first image cluster, and clustering the images in the second image set to obtain at least one second image cluster, comprises: obtaining a first feature of each first image and an initial second feature of each second image; Determine each nearest neighboring first image pair based on the first feature of each first image, and determine each nearest neighboring second image pair based on the initial second feature of each second image, wherein the two first images in each nearest neighboring first image pair are nearest neighbors to each other; and the two second images in each nearest neighboring second image pair are nearest neighbors to each other; updating the nearest second image pair where the second image corresponding to the two first images is located according to the correspondence between the two first images in each nearest first image pair; For each second image, taking a nearest image in all updated nearest second image pairs of the second image as a target nearest image, and obtaining a target second feature of the second image according to an initial second feature of the target nearest image; Clustering is performed on the first features of all first images to obtain at least one first image cluster; and clustering is performed on the target second features of all second images to obtain at least one second image cluster.

11. The method according to claim 10, characterized in that Updating the nearest second image pair where the second images corresponding to the two first images are located includes: For each nearest first image pair, two first images in the nearest first image pair are determined; if both first images have corresponding target second images, the two corresponding target second images are determined as the nearest second image pair.

12. A video processing method, characterized in that: include: Detecting a first part and a second part of a target object on at least a portion of video frames in a target video to obtain a first image set and a second image set; Each first image in the first image set is a subregion of a first part of a target object in a video frame; each second image in the second image set is a subregion of a second part of a target object in a video frame; dividing the audio corresponding to the target video into a plurality of audio segments based on characters, and clustering the plurality of audio segments to obtain at least one audio segment cluster; Clustering the images in the first image set to obtain at least one first image cluster, clustering the images in the second image set to obtain at least one second image cluster, and assigning an object label to each of the first image clusters; the detection accuracy of the first part is high; For each audio segment cluster, determining a matching degree between the audio segment cluster and each to-be-matched image cluster, wherein the to-be-matched image cluster includes at least one of the first image cluster or the second image cluster; For any first image cluster, an audio clip cluster whose matching degree with the first image cluster meets a second preset condition is used as a reference audio cluster, a second image cluster whose matching degree with the reference audio cluster meets the second preset condition is used as a reference second image cluster, and a target video frame corresponding to the reference audio cluster and an object label of the reference second image cluster are determined as the object label of the first image cluster; According to each first image cluster, the second image cluster with determined object labels, and the object labels corresponding to the target video frame, a target video segment corresponding to each object label is determined from the target video.

13. The video processing method according to claim 12, wherein: The detecting of the first part and the second part of the target object on at least a portion of the video frames in the target video further includes: Obtaining a reference video frame set, wherein each reference video frame in the reference video frame set simultaneously contains at least one first image and at least one second image; The method further comprises: For each of the reference video frames, determining a correspondence between a first image and a second image belonging to the same target object; For each second image cluster, the object label of the second image cluster is determined according to the object label of the first image corresponding to the matching image in the second image cluster; the matching image is the second image in the second image cluster having the corresponding relationship.

14. A video processing device, characterized in that: include: A multi-mode detection module is configured to detect a first portion and a second portion of a target object in at least a portion of video frames in a target video, thereby obtaining a first image set, a second image set, and a reference video frame set; each first image in the first image set is a subregion of the first portion of a target object in a video frame; each second image in the second image set is a subregion of the second portion of a target object in a video frame; each reference video frame in the reference video frame set simultaneously contains at least one first image and at least one second image; and the detection accuracy of the first portion is high. a correspondence determination module, configured to determine, for each reference video frame, a correspondence between a first image and a second image belonging to the same target object; a clustering module, configured to cluster the images in the first image set to obtain at least one first image cluster, cluster the images in the second image set to obtain at least one second image cluster, and set an object label for each of the first image clusters; a label determination module configured to determine, for each second image cluster, an object label of the second image cluster based on an object label of a first image corresponding to a matching image in the second image cluster; the matching image being a second image in the second image cluster having the corresponding relationship; The video highlights module is used to determine target video segments corresponding to each object label from the target video according to the object labels corresponding to each first image cluster and the second image cluster with determined object labels.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images