Video processing method, device and electronic equipment

Abstract

The present invention provides a video processing method, device and electronic device, which relate to the field of video processing technology. The method comprises: extracting at least one video frame set from a video to be processed; wherein each of the video frame sets comprises at least one video frame; extracting image features of the video frames in each video frame set; based on the image features and the object features of a preset target object, screening out a target video frame set that matches the target object from the at least one video frame set; and obtaining a processing result according to the video frames corresponding to the target video frame set in the video to be processed. The embodiment of the present invention pre-splits the video to be processed into a plurality of sub-strips by using target detection and other technologies, and then reorganizes the sub-strips according to the face dimension information between different sub-strips in combination with face recognition and other technologies, thereby realizing automated video stripping, improving stripping efficiency and reducing labor costs.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video processing method, device and electronic equipment. Background Art

[0002] Video stripping is a technical method of secondary processing of traditional video programs and media data, splitting multiple complete video data according to specific logic and needs, and re-arranging them, such as "highlights of so-and-so" and "only watch him on the video website". Video stripping is of great significance in the fields of video intelligent analysis, media asset management, and Internet short video platforms.

[0003] At present, the implementation of video stripping mostly relies on manual processing, which involves manually previewing, splitting and combining videos. This method has the following shortcomings: first, the labor cost is high and the processing time is long; second, with the continuous growth of video scale, video stripping has changed from a traditional video processing task to a big data computing task, and the manual video stripping method cannot meet the stripping needs of large-scale videos; third, the efficiency of manual video stripping is low and cannot meet the real-time requirements of stripping for popular videos or videos during important periods.

[0004] In general, the existing method of manually splitting videos has high processing costs, low processing efficiency, and poor real-time performance. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a video processing method, device and electronic equipment, which realize automatic video stripping, reduce the cost of video stripping, improve processing efficiency, and have better real-time performance.

[0006] In a first aspect, an embodiment of the present invention provides a video processing method, comprising: extracting at least one video frame set from a video to be processed; wherein each of the video frame sets includes at least one video frame; extracting image features of the video frames in each video frame set; based on the image features and object features of a preset target object, screening out a target video frame set that matches the target object from the at least one video frame set; and obtaining a processing result based on the video frames corresponding to the target video frame set in the video to be processed.

[0007] In an optional embodiment of the present invention, the above-mentioned step of extracting at least one video frame set from the video to be processed includes: extracting frames from the video to be processed at a preset time interval to obtain multiple video frames; and dividing the multiple video frames into at least one video frame set.

[0008] In an optional embodiment of the present invention, the above-mentioned step of extracting frames from the video to be processed at preset time intervals to obtain multiple video frames includes: extracting frames from the video to be processed at preset time intervals to obtain multiple video frames through a preset multimedia video processing tool.

[0009] In an optional embodiment of the present invention, the step of dividing the multiple video frames into at least one video frame set includes: dividing the multiple video frames into at least one video frame set according to similarities between adjacent video frames in the multiple video frames.

[0010] In an optional embodiment of the present invention, the step of dividing the multiple video frames into at least one video frame set according to the similarity between adjacent video frames in the multiple video frames includes: selecting video frames in sequence from the multiple video frames, determining the selected video frames as current video frames, and performing the following processing on the current video frame: calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame; if the similarity value is greater than a preset first threshold, dividing the current video frame and the next video frame into the same video frame set; determining the next video frame of the current video frame as a new current video frame, and continuing to perform the step of calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame, and if the similarity value is greater than the preset first threshold, dividing the current video frame and the next video frame into the same video frame set, until the last video frame of the multiple video frames, to obtain at least one video frame set.

[0011] In an optional embodiment of the present invention, the above-mentioned step of calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame includes: extracting a first image feature of the current video frame and a second image feature of the next video frame adjacent to the current video frame; calculating a cosine distance between the first image feature and the second image feature; and determining the similarity value between the current video frame and the next video frame based on the cosine distance.

[0012] In an optional embodiment of the present invention, the above-mentioned step of extracting the first image feature of the current video frame and the second image feature of the next video frame adjacent to the current video frame includes: extracting the first image feature of the current video frame and the second image feature of the next video frame adjacent to the current video frame according to a scale-invariant feature transform algorithm (SIFT).

[0013] In an optional embodiment of the present invention, the above-mentioned step of extracting image features of each video frame in the video frame set includes: extracting image features of each video frame in the video frame set through a deep convolutional neural network.

[0014] In an optional embodiment of the present invention, the step of filtering out a target video frame set that matches the target object from the at least one video frame set based on the image features and the object features of the preset target object comprises: for each video frame set in the at least one video frame set, traversing each video frame in the video frame set, and performing the following processing on each video frame: determining whether the video frame contains the target object based on the image features of the video frame and the object features of the preset target object; if so, determining a preset label of the target object as the label of the video frame; calculating the ratio of each type of label in the video frame set to the total number of labels in the video frame set; and filtering out the target video frame set from the at least one video frame set based on the ratio.

[0015] In an optional embodiment of the present invention, the step of determining whether the target object is contained in the video frame based on the image features of the video frame and the object features of a preset target object comprises: based on the image features of the video frame, retrieving object features matching the image features from a preset database; wherein the object features of the target object are pre-stored in the database; if object features matching the image features are retrieved, determining that the target object is contained in the video frame.

[0016] In an optional embodiment of the present invention, the step of retrieving object features that match the image features from a preset database based on the image features of the video frame includes: calculating the cosine similarity between the image features of the video frame and each object feature in the preset database; if the cosine similarity is greater than a preset second threshold, determining the object feature corresponding to the cosine similarity as the object feature that matches the image feature.

[0017] In an optional embodiment of the present invention, the method further includes: if the target object is not included in the video frame, determining that the label of the video frame is a preset designated label.

[0018] In an optional embodiment of the present invention, the step of filtering out a target video frame set from the at least one video frame set based on the ratio includes: if the ratio is greater than a preset third threshold, determining the video frame set corresponding to the ratio as the target video frame set.

[0019] In an optional embodiment of the present invention, the step of obtaining the processing result based on the video frames corresponding to the target video frame set in the video to be processed includes: determining the target video segment based on the timestamps of the video frames corresponding to the target video frame set in the video to be processed; and outputting the target video segment as the processing result.

[0020] In an optional embodiment of the present invention, there are multiple target video segments, and the step of outputting the target video segment as a processing result includes: aggregating video segments matching the same target object among the multiple target video segments; and outputting the aggregated and reorganized video segments as the processing result.

[0021] In an optional embodiment of the present invention, the target object is a designated person.

[0022] In an optional embodiment of the present invention, the above method is applied to a Spark big data processing platform. Before the step of extracting at least one set of video frames from the video to be processed, the method also includes: converting the video format of the video to be processed into a resilient distributed dataset (RDD) format.

[0023] In a second aspect, an embodiment of the present invention further provides a video processing device, comprising: a video frame set extraction module, used to extract at least one video frame set from a video to be processed; wherein each of the video frame sets includes at least one video frame; an image feature extraction module, used to extract image features of video frames in each of the video frame sets; a target video frame set screening module, used to screen out a target video frame set matching the target object from the at least one video frame set based on the image features and the object features of a preset target object; and a processing result module, used to obtain a processing result based on the video frames corresponding to the target video frame set in the video to be processed.

[0024] In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the above-mentioned video processing method.

[0025] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned video processing method.

[0026] The embodiments of the present invention bring the following beneficial effects:

[0027] The embodiment of the present invention provides a video processing method, device and electronic device, which extracts at least one video frame set from a video to be processed; wherein each of the video frame sets includes at least one video frame; extracts the image features of the video frames in each video frame set; based on the image features and the object features of the preset target object, a target video frame set matching the target object is screened out from the at least one video frame set; and a processing result is obtained according to the video frames corresponding to the target video frame set in the video to be processed. In this method, the video to be processed is pre-split into multiple sub-strips by target detection and other technologies, and then the sub-strips are reorganized according to the face dimension information between different sub-strips, combined with face recognition and other technologies, to realize automatic video stripping, improve stripping efficiency and reduce labor costs; in addition, the present application also realizes distributed stripping based on a distributed computing platform, improves the efficiency of video frame extraction, video stripping, feature extraction and retrieval query by parallel computing, further improves the efficiency of video stripping, and has better real-time stripping.

[0028] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0029] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present invention;

[0032] Figure 2 A schematic flow chart of another video processing method provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a distributed video stripping process provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of the structure of a video processing device provided by an embodiment of the present invention;

[0035] Figure 5A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0036] Icons: 41 - video frame set extraction module; 42 - image feature extraction module; 43 - target video frame set screening module; 44 - processing result module; 51 - processor; 52 - memory; 53 - bus; 54 - communication interface. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Considering the existing manual video stripping method, which has high processing cost, low processing efficiency and poor real-time performance, the embodiments of the present invention provide a video processing method, device and electronic device, which can be applied to video stripping, re-editing and large-scale video stripping. To facilitate understanding of this embodiment, a video processing method disclosed in the embodiment of the present invention is first introduced in detail.

[0039] See also Figure 1 , which is a flow chart of a video processing method. Figure 1 It can be seen that the video processing method includes the following steps:

[0040] Step S102: extracting at least one video frame set from the video to be processed; wherein each of the video frame sets includes at least one video frame.

[0041] Here, the video to be processed may be in any video format, such as MPEG, MPG, DAT, MP4, etc. The above video frame set is composed of video frames extracted from the video to be processed. In one possible implementation, at least one video frame set may be extracted from the video to be processed by the following steps 11-12:

[0042] (11) Extract frames from the video to be processed at a preset time interval to obtain multiple video frames.

[0043] In actual operation, a preset multimedia video processing tool can be used to extract frames from the video to be processed at preset time intervals to obtain multiple video frames. The multimedia video processing tool can be FFmpeg, MediaStudio Pro, Adobe Premiere, etc. Taking FFmpeg as an example, the video to be processed can be extracted once every 5 seconds to obtain multiple video frames.

[0044] (12) Divide the multiple video frames into at least one video frame set.

[0045] In at least one possible implementation, the multiple video frames can be divided into at least one video frame set according to the similarity between adjacent video frames in the multiple video frames. Specifically, video frames can be selected from the multiple video frames in sequence, the selected video frames are determined as the current video frames, and the current video frames are processed as follows: first, the similarity value between the current video frame and the next video frame adjacent to the current video frame is calculated; wherein, if the similarity value is greater than a preset first threshold, the current video frame and the next video frame are divided into the same video frame set; then, the next video frame of the current video frame is determined as the new current video frame, and the step of calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame is continued, and if the similarity value is greater than the preset first threshold, the current video frame and the next video frame are divided into the same video frame set, until the last video frame in the multiple video frames, to obtain at least one video frame set.

[0046] Here, when calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame, the following steps 21-23 are performed:

[0047] (21) Extracting a first image feature of the current video frame and a second image feature of a next video frame adjacent to the current video frame.

[0048] In one possible implementation, the first image feature of the current video frame and the second image feature of the next video frame adjacent to the current video frame can be extracted according to the scale-invariant feature transform (SIFT) algorithm. Here, the scale-invariant feature transform algorithm is a description used in the field of image processing. This description has scale invariance and can detect key points in the image. In this embodiment, the SIFT algorithm can quickly distinguish whether two adjacent video frames are in the same video scene, thereby quickly dividing the video frames into scenes. For example, for a video frame with a snowy background and a video frame with an indoor background, the SIFT algorithm can quickly determine that the scenes of the two are different, thereby dividing the two into different sets of video frames.

[0049] (22) Calculate the cosine distance between the first image feature and the second image feature.

[0050] Here, the cosine distance is also the cosine similarity, and the similarity between the first image feature and the second image feature is evaluated by calculating the cosine value of the angle between the two vectors.

[0051] (23) Determine a similarity value between the current video frame and the next video frame according to the cosine distance.

[0052] Among them, the smaller the cosine distance is, the more similar the two video frames are. Conversely, the larger the cosine distance is, the smaller the similarity value between the two is, and the worse the similarity is.

[0053] Step S104: extracting image features of the video frames in each video frame set.

[0054] In actual operation, the image features of each video frame in the video frame set can be extracted through deep convolutional neural networks (CNN).

[0055] Step S106: based on the image feature and the object feature of the preset target object, a target video frame set matching the target object is screened out from the at least one video frame set.

[0056] Here, the target object may be a designated person, animal, building or other object. Taking the designated person as an example, for example, the preset target object is a star A. By obtaining the features of the star A in advance, combining the various video frame sets extracted from the video to be processed in the above steps, and the image features of the video frames in each video frame set, assuming that the image features of the video frames contained in a certain video frame set M are similar or identical to the features of the star A, the video frame set is considered to match the target object, and thus is determined to be the target video frame set.

[0057] In other possible implementations, other matching rules may be set. When the image features of the video frames in the video frame set and the object features of the preset target object meet the matching rules, the video frame set is considered to be a target video frame set that matches the target object.

[0058] Step S108: obtaining a processing result according to the video frames corresponding to the target video frame set in the video to be processed.

[0059] In actual operation, assuming that it is necessary to split the video to be processed to obtain the required video segments, for example, to separate the video segment in which a star A appears from an entertainment video, the target video frame set including the star A can be obtained through the aforementioned steps S102 to S106. However, the target video frame set is obtained by extracting frames from the video to be processed, and its time is often discontinuous. Therefore, it is necessary to determine the time range of the target video segment according to the time of the first video frame in the target video frame set and the time of the last video frame, so as to obtain the video segment containing the star A, that is, the final required processing result.

[0060] Here, in the implementation shown in the above example, the target video segment is determined according to the timestamp of the video frame corresponding to the target video frame set in the video to be processed; and then the target video segment is output as the processing result.

[0061] In actual operation, suppose that the video segments of star A and star B need to be separated from the video to be processed, wherein star A may have appeared in different scenes in the above-mentioned video to be processed, for example, in the beginning and the end of the film. In this way, after the above steps, multiple video frame sets may be obtained, all of which contain the video frames of star A, and then multiple corresponding video segments containing star A are obtained. In one possible implementation, when encountering the above scenario, the video segments matching the same target object in the multiple target video segments can be aggregated; and then the aggregated and reorganized video segments are output as the processing results. In this way, after aggregation and reorganization, the video segments about star A can be aggregated and output, and the video segments about star B can also be aggregated and output, which is more in line with actual needs.

[0062] A video processing method provided by an embodiment of the present invention extracts at least one video frame set from a video to be processed; wherein each of the video frame sets includes at least one video frame; extracts image features of the video frames in each video frame set; based on the image features and the object features of a preset target object, selects a target video frame set that matches the target object from the at least one video frame set; and obtains a processing result based on the video frames corresponding to the target video frame set in the video to be processed. In this method, the video to be processed is pre-split into multiple sub-strips by target detection and other technologies, and then the sub-strips are reorganized based on the face dimension information between different sub-strips in combination with face recognition and other technologies, thereby realizing automated video stripping, improving stripping efficiency, and reducing labor costs.

[0063] exist Figure 1 Based on the video processing method shown, an embodiment of the present invention also provides another video processing method, which focuses on describing the specific implementation process of step S106 in the aforementioned embodiment (based on the image features and the object features of the preset target object, screening out a target video frame set that matches the target object from the above-mentioned at least one video frame set).

[0064] like Figure 2 As shown, it is a flow chart of the video processing method. Figure 2 It can be seen that the method comprises the following steps:

[0065] Step S202: extracting at least one video frame set from the video to be processed; wherein each of the video frame sets includes at least one video frame.

[0066] Step S204: extracting image features of the video frames in each video frame set.

[0067] Here, steps S202 to S204 in this embodiment correspond to steps S102 to S104 in the above embodiment. For the description of the corresponding contents, reference may be made to the corresponding parts of the above embodiment and will not be repeated here.

[0068] Step S206: For each video frame set in the at least one video frame set, traverse each video frame in the video frame set, and perform the following processing on each video frame: based on the image features of the video frame and the object features of the preset target object, determine whether the video frame contains the target object; if so, determine the preset label of the target object as the label of the video frame.

[0069] Here, the step of determining whether the video frame contains the target object based on the image features of the video frame and the object features of the preset target object can be implemented by the following steps 31-32:

[0070] (31) Based on the image features of the video frame, object features matching the image features are retrieved from a preset database; wherein the object features of the target object are pre-stored in the database.

[0071] Specifically, the cosine similarity between the image feature of the video frame and each object feature in the preset database can be calculated; if the cosine similarity is greater than a preset second threshold, the object feature corresponding to the cosine similarity is determined as the object feature matching the image feature.

[0072] For example, assuming that the target object is star A, the character features of star A are obtained in advance, such as by collecting various picture materials about star A, extracting the features of star A in these pictures, including but not limited to its face shape, facial features, body shape, etc., and saving the above features of star A to a preset database, so that when performing retrieval and matching, similarity retrieval can be performed based on the preset database. Among them, assuming that the preset second threshold is 85%, in the retrieval, the object features in the database whose cosine similarity with the image features of the video frame is greater than 85% are all determined as object features that match the image features.

[0073] (32) If an object feature matching the image feature is retrieved, it is determined that the target object is contained in the video frame.

[0074] If it is determined that the target object is included in the video frame, the preset label of the target object is determined as the label of the video frame. For example, if it is determined that the video frame contains star A, the name of star A can be used as the label of the video frame. In this way, according to the above steps, the label of each video frame in the video frame set can be obtained.

[0075] In actual operation, if the target object is not included in the video frame, the label of the video frame is determined to be a preset designated label. Here, the designated label can be "NONE" or other labels that will not cause confusion, which is not limited here.

[0076] Step S208: Calculate the ratio of each type of label in the video frame set to the total number of labels in the video frame set.

[0077] Here, we still take the target object as a star as an example. Assume that the target object includes multiple stars, such as star A, star B and star C. In a certain video frame set extracted above, if the video frame set has 100 video frames, after the above steps S202 to S206, the corresponding label has been determined for each video frame in the video frame set, among which 10 video frames correspond to the label of star A, 40 video frames correspond to the label of star B, 20 video frames correspond to the label of star C, and 30 video frames correspond to the label of NONE (that is, the above star A or star B or star C is detected in these 30 video frames). In this way, it can be seen that the video frame set includes a total of four types of labels: star A, star B, star C and NONE, wherein the ratio of each type of label to the total number of labels in the video frame set is calculated as follows: star A accounts for 10%, star B accounts for 40%, star C accounts for 20%, and NONE accounts for 30%.

[0078] Step S210: Filtering a target video frame set from the at least one video frame set according to the ratio.

[0079] Here, in at least one possible implementation, if the ratio is greater than a preset third threshold, the video frame set corresponding to the ratio is determined as the target video frame set.

[0080] For example, assuming that the third threshold value is set to 50%, for the above example, the highest percentage is Star B, but its percentage is still lower than the preset threshold value of 50%, so the video frame set is discarded and not output; if the third threshold value is set to 35%, the percentage of Star B's label meets the requirement, so the video frame set is retained, and the corresponding video segment is determined based on the video frame set, and the label "Star B" is used as the label of the output video segment and output together. Here, one of the output formats can be the key-value format, for example, the output is<face_id,video_id> , where face_id represents the face tag, which is the key; video_id represents the ID of the output video segment, that is, the ID of the video sub-strip, which is the corresponding value.

[0081] Step S212: obtaining a processing result according to the video frames corresponding to the target video frame set in the video to be processed.

[0082] Here, step S212 of this embodiment corresponds to step S108 of the aforementioned embodiment. For the description of the corresponding contents, reference may be made to the corresponding parts of the aforementioned embodiment, which will not be repeated here.

[0083] In another possible implementation, in order to improve the speed of video processing, the video processing method provided in this embodiment can also be applied to a distributed processing platform to implement distributed video striping. Figure 3 , which is a schematic diagram of a distributed video stripping process. Figure 3 In the illustrated implementation, the video processing method is applied to the Spark big data processing platform, wherein, before the video to be processed is sent to each processing node, the video format of the video to be processed needs to be converted into a resilient distributed dataset (RDD) format, and the calculation of each node is based on the video file RDD. Specifically, the method simultaneously executes the aforementioned steps S202 to S210 on the video to be processed on multiple distributed nodes to realize video stripping of different target objects and obtain corresponding multiple target video segments (i.e., target video sub-stripping).

[0084] In actual operation, based on Figure 3 The distributed video stripping method, whose distributed computing is mainly divided into two stages, one is the Map stage, and the other is the Reduce stage. Among them, in the Map stage, each processing node performs video frame extraction, obtains video sub-strips, extracts facial features, searches the facial database, and generates sub-strip labels; in the Reduce stage, the sub-strips output by each processing node are aggregated, and the sub-strips with the same facial labels are aggregated together, and finally the aggregated processing results are output. It should be noted that the above-mentioned facial database is an image library established according to the specified face, and the above-mentioned video sub-strips are a set of video frames composed of video frames obtained by video frame extraction. In this way, based on the above steps, the video is split by face.

[0085] The video processing method provided in this embodiment, on the one hand, realizes automatic video stripping through intelligent technologies such as target detection and face recognition; on the other hand, it realizes distributed stripping based on a distributed computing platform, and improves the efficiency of video frame extraction, video stripping, feature extraction and retrieval query through parallel computing, further improving the efficiency of video stripping, with better real-time stripping, which can effectively alleviate the problem of long video stripping time and low efficiency caused by the increase in data scale.

[0086] Corresponds to Figure 1 The video processing method shown in the embodiment of the present invention also provides a video processing device, such as Figure 4 As shown, it is a schematic diagram of the structure of the video processing device. Figure 4 It can be seen that the device includes a video frame set extraction module 41, an image feature extraction module 42, a target video frame set screening module 43 and a processing result module 44 which are connected in sequence, wherein the functions of each module are as follows:

[0087] The video frame set extraction module 41 is used to extract at least one video frame set from the video to be processed; wherein each of the video frame sets includes at least one video frame;

[0088] An image feature extraction module 42, used to extract image features of each video frame in the video frame set;

[0089] A target video frame set screening module 43 is used to screen a target video frame set matching the target object from the at least one video frame set based on the image feature and the object feature of the preset target object;

[0090] The processing result module 44 is used to obtain a processing result according to the video frame corresponding to the target video frame set in the video to be processed.

[0091] The embodiment of the present invention provides a video processing device, which extracts at least one video frame set from a video to be processed; wherein each of the video frame sets includes at least one video frame; extracts image features of the video frames in each video frame set; based on the image features and the object features of the preset target object, a target video frame set matching the target object is screened out from the at least one video frame set; and a processing result is obtained according to the video frames corresponding to the target video frame set in the video to be processed. In the device, the video to be processed is pre-split into multiple sub-strips by target detection and other technologies, and then the sub-strips are reorganized according to the face dimension information between different sub-strips in combination with face recognition and other technologies, thereby realizing automatic video stripping, improving stripping efficiency, and reducing labor costs.

[0092] In one possible implementation manner, the video frame set extraction module 41 is further used to: extract frames from the video to be processed at a preset time interval to obtain multiple video frames; and divide the multiple video frames into at least one video frame set.

[0093] In another possible implementation, the video frame set extraction module 41 is further used to extract frames from the video to be processed at preset time intervals using a preset multimedia video processing tool to obtain multiple video frames.

[0094] In another possible implementation, the video frame set extraction module 41 is further configured to: divide the multiple video frames into at least one video frame set according to similarities between adjacent video frames in the multiple video frames.

[0095] In another possible implementation, the video frame set extraction module 41 is further used to: select video frames in sequence from the multiple video frames, determine the selected video frame as the current video frame, and perform the following processing on the current video frame: calculate the similarity value between the current video frame and the next video frame adjacent to the current video frame; if the similarity value is greater than a preset first threshold, divide the current video frame and the next video frame into the same video frame set; determine the next video frame of the current video frame as the new current video frame, and continue to execute the step of calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame; if the similarity value is greater than the preset first threshold, divide the current video frame and the next video frame into the same video frame set, until the last video frame of the multiple video frames, to obtain at least one video frame set.

[0096] In another possible implementation, the video frame set extraction module 41 is further used to: extract a first image feature of the current video frame and a second image feature of a next video frame adjacent to the current video frame; calculate a cosine distance between the first image feature and the second image feature; and determine a similarity value between the current video frame and the next video frame based on the cosine distance.

[0097] In another possible implementation, the video frame set extraction module 41 is further used to extract the first image feature of the current video frame and the second image feature of the next video frame adjacent to the current video frame according to a scale-invariant feature transform (SIFT) algorithm.

[0098] In another possible implementation, the image feature extraction module 42 is further used to extract image features of each video frame in the video frame set through a deep convolutional neural network.

[0099] In another possible implementation, the target video frame set screening module 43 is further used to: for each video frame set in the at least one video frame set, traverse each video frame in the video frame set, and perform the following processing on each video frame: determine whether the video frame contains the target object based on the image features of the video frame and the object features of the preset target object; if so, determine the preset label of the target object as the label of the video frame; calculate the ratio of each type of label in the video frame set to the total number of labels in the video frame set; and based on the ratio, screen out the target video frame set from the at least one video frame set.

[0100] In another possible implementation, the target video frame set screening module 43 is further used to: based on the image features of the video frame, retrieve object features that match the image features from a preset database; wherein the database pre-stores the object features of the target object; if an object feature that matches the image features is retrieved, it is determined that the target object is contained in the video frame.

[0101] In another possible implementation, the target video frame set screening module 43 is further used to calculate the cosine similarity between the image feature of the video frame and each object feature in a preset database; if the cosine similarity is greater than a preset second threshold, the object feature corresponding to the cosine similarity is determined as the object feature that matches the image feature.

[0102] In another possible implementation, the target video frame set screening module 43 is further configured to: if the video frame does not contain the target object, determine that the label of the video frame is a preset designated label.

[0103] In another possible implementation, the target video frame set screening module 43 is further configured to: if the ratio is greater than a preset third threshold, determine the video frame set corresponding to the ratio as the target video frame set.

[0104] In another possible implementation, the processing result module 44 is further used to: determine a target video segment according to the timestamp of a video frame corresponding to the target video frame set in the video to be processed; and output the target video segment as a processing result.

[0105] In another possible implementation, there are multiple target video segments, and the processing result module 44 is further used to: aggregate video segments matching the same target object among the multiple target video segments; and output the aggregated and reorganized video segments as processing results.

[0106] In another possible implementation, the target object is a designated person.

[0107] In another possible implementation, the above-mentioned device is applied to a Spark big data processing platform, and the device also includes a format conversion module for converting the video format of the video to be processed into a resilient distributed dataset (RDD) format.

[0108] The video processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned video processing method embodiment. For the sake of brief description, for parts not mentioned in the embodiment of the video processing device, reference may be made to the corresponding contents in the aforementioned video processing method embodiment.

[0109] The embodiment of the present invention further provides an electronic device, such as Figure 5As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 51 and a memory 52, the memory 52 stores machine executable instructions that can be executed by the processor 51, and the processor 51 executes the machine executable instructions to implement the above-mentioned video processing method.

[0110] exist Figure 5 In the illustrated embodiment, the electronic device further includes a bus 53 and a communication interface 54 , wherein the processor 51 , the communication interface 54 and the memory 52 are connected via the bus.

[0111] The memory 52 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 54 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0112] The processor 51 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 51. The above processor 51 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 51 reads the information in the memory 52 and completes the steps of the video processing method of the above embodiment in combination with its hardware.

[0113] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned video processing method. The specific implementation can be referred to the aforementioned method embodiment, which will not be repeated here.

[0114] The video processing method, video processing device and computer program product of the electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the video processing method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0115] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0116] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0117] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0118] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: include: Extracting at least one video frame set from the video to be processed; wherein each of the video frame sets includes at least one video frame; Extracting image features of video frames in each of the video frame sets; Based on the image features and the object features of the preset target object, a target video frame set matching the target object is screened out from the at least one video frame set; Obtaining a processing result according to the video frames corresponding to the target video frame set in the video to be processed; The step of extracting at least one video frame set from the video to be processed includes: extracting frames from the video to be processed at preset time intervals through a preset multimedia video processing tool to obtain multiple video frames; selecting video frames from the multiple video frames in sequence, determining the selected video frames as current video frames, and performing the following processing on the current video frame: calculating a similarity value between the current video frame and a next video frame adjacent to the current video frame; if the similarity value is greater than a preset first threshold, classifying the current video frame and the next video frame into the same video frame set; determining the next video frame of the current video frame as a new current video frame, and continuing to perform the step of calculating a similarity value between the current video frame and a next video frame adjacent to the current video frame, and if the similarity value is greater than the preset first threshold, classifying the current video frame and the next video frame into the same video frame set, until the last video frame of the multiple video frames, to obtain at least one video frame set; The step of filtering out a target video frame set matching the target object from the at least one video frame set based on the image features and the object features of the preset target object comprises: for each video frame set in the at least one video frame set, traversing each video frame in the video frame set, and performing the following processing on each video frame: determining whether the video frame contains the target object based on the image features of the video frame and the object features of the preset target object; if so, determining the preset label of the target object as the label of the video frame; calculating the ratio of each type of label in the video frame set to the total number of labels in the video frame set; and filtering out the target video frame set from the at least one video frame set according to the ratio; The step of obtaining a processing result based on the video frames corresponding to the target video frame set in the video to be processed includes: determining a target video segment based on the timestamps of the video frames corresponding to the target video frame set in the video to be processed; aggregating video segments that match the same target object among multiple target video segments when there are multiple target video segments; and outputting the aggregated and reorganized video segments as the processing result.

2. The video processing method according to claim 1, characterized in that: The step of calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame comprises: Extracting a first image feature of the current video frame and a second image feature of a next video frame adjacent to the current video frame; calculating a cosine distance between the first image feature and the second image feature; A similarity value between the current video frame and the next video frame is determined according to the cosine distance.

3. The video processing method according to claim 2, characterized in that: The step of extracting the first image feature of the current video frame and the second image feature of the next video frame adjacent to the current video frame comprises: According to a scale-invariant feature transformation algorithm, a first image feature of the current video frame and a second image feature of a next video frame adjacent to the current video frame are extracted.

4. The video processing method according to claim 1, characterized in that: The step of extracting image features of each video frame in the video frame set comprises: The image features of the video frames in each of the video frame sets are extracted through a deep convolutional neural network.

5. The video processing method according to claim 1, characterized in that: The step of determining whether the video frame contains the target object based on the image features of the video frame and the object features of the preset target object comprises: Based on the image features of the video frame, an object feature matching the image features is retrieved from a preset database; wherein the object features of the target object are pre-stored in the database; If an object feature matching the image feature is retrieved, it is determined that the target object is contained in the video frame.

6. The video processing method according to claim 5, characterized in that: The step of retrieving object features matching the image features from a preset database based on the image features of the video frame comprises: Calculate the cosine similarity between the image feature of the video frame and the feature of each object in the preset database; If the cosine similarity is greater than a preset second threshold, the object feature corresponding to the cosine similarity is determined as the object feature matching the image feature.

7. The video processing method according to claim 1, characterized in that: The method further comprises: If the target object is not included in the video frame, it is determined that the label of the video frame is a preset designated label.

8. The video processing method according to claim 1, characterized in that: The step of selecting a target video frame set from the at least one video frame set according to the ratio comprises: If the ratio is greater than a preset third threshold, the video frame set corresponding to the ratio is determined as the target video frame set.

9. The video processing method according to claim 1, characterized in that: The target object is a designated person.

10. The video processing method according to any one of claims 1 to 9, characterized in that: The method is applied to the Spark big data processing platform. Before the step of extracting at least one video frame set from the video to be processed, the method further includes: Convert the video format of the video to be processed into the elastic distributed dataset format.

11. A video processing device, characterized in that: include: A video frame set extraction module, used to extract at least one video frame set from the video to be processed; wherein each of the video frame sets includes at least one video frame; An image feature extraction module, used to extract image features of video frames in each of the video frame sets; a target video frame set screening module, configured to screen a target video frame set matching the target object from the at least one video frame set based on the image features and the object features of the preset target object; A processing result module, used for obtaining a processing result according to the video frame corresponding to the target video frame set in the video to be processed; The video frame set extraction module is used to extract frames from the video to be processed at preset time intervals through a preset multimedia video processing tool to obtain multiple video frames; select video frames from the multiple video frames in sequence, determine the selected video frames as current video frames, and perform the following processing on the current video frame: calculate the similarity value between the current video frame and the next video frame adjacent to the current video frame; if the similarity value is greater than a preset first threshold, classify the current video frame and the next video frame into the same video frame set; determine the next video frame of the current video frame as a new current video frame, and continue to perform the steps of calculating the similarity value between the current video frame and the next video frame adjacent to the current video frame, and if the similarity value is greater than the preset first threshold, classify the current video frame and the next video frame into the same video frame set, until the last video frame of the multiple video frames, to obtain at least one video frame set; The target video frame set screening module is used for traversing each video frame in the video frame set for each video frame set in the at least one video frame set, and performing the following processing on each video frame: determining whether the video frame contains the target object based on the image features of the video frame and the object features of the preset target object; if so, determining the preset label of the target object as the label of the video frame; calculating the ratio of each type of label in the video frame set to the total number of labels in the video frame set; and screening out the target video frame set from the at least one video frame set according to the ratio; The processing result module is used to determine the target video segment according to the timestamp of the video frame corresponding to the target video frame set in the video to be processed; if there are multiple target video segments, the video segments matching the same target object in the multiple target video segments are aggregated; and the aggregated and reorganized video segments are output as processing results.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the video processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the video processing method according to any one of claims 1 to 10.

Images