Video processing method, video processing device and terminal equipment

Abstract

The present invention is applicable to the field of image processing technology, and provides a video processing method, a video processing device and a terminal device. The video processing method includes: obtaining a video to be processed; determining the scene category of each video frame in the video to be processed; according to the scene category of each video frame, performing target recognition on each video frame in the video to be processed by a trained first deep learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category; according to the target recognition result, identifying multiple target frames of the video to be processed, wherein the target frame contains the target object; according to the time sequence of each target frame in the video to be processed, sequentially combining the multiple target frames to obtain the target video.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a video processing method, a video processing device and a terminal device. Background Art

[0002] At present, video image technology is widely used in many fields (such as video surveillance, mobile terminals, and social platforms, etc.), and a large number of video files are generated during the application process. These video files often contain a large number of redundant or unimportant parts, and the duration of these video files is often very long, resulting in users needing to spend a long time to query the required key information from the video files. For example, a user needs to query the video part of a certain individual performing a specified activity from a surveillance video with a duration of 24 hours. At present, the method that users can take is to fast-forward the video file and mark the time corresponding to the video part of a certain individual performing a specified activity, so as to find the required video part from the video file according to the corresponding time in subsequent use. However, due to the existence of a large number of redundant or unimportant parts in the video file, this method of querying key information in the video is time-consuming, and it is difficult for users to efficiently extract some key information in the video. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a video processing method, a video processing device and a terminal device, which can identify and extract key information in a video (such as identifying and extracting a video portion containing a target individual), thereby improving the efficiency of users in obtaining key information in a video.

[0004] A first aspect of an embodiment of the present invention provides a video processing method, including:

[0005] Get the video to be processed;

[0006] Determining a scene category of each video frame in the video to be processed;

[0007] According to the scene category of each video frame, target recognition is performed on each video frame in the to-be-processed video by using the trained first deep learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category;

[0008] According to the target recognition result, identifying a plurality of target frames of the video to be processed, wherein the target frame contains the target object;

[0009] According to the time sequence of each target frame in the video to be processed, multiple target frames are combined in sequence to obtain the target video.

[0010] A second aspect of an embodiment of the present invention provides a video processing device, including:

[0011] An acquisition module is used to acquire the video to be processed;

[0012] A determination module, used to determine the scene category of each video frame in the video to be processed;

[0013] A first recognition module is used to perform target recognition on each video frame in the to-be-processed video by using a trained first deep learning model according to the scene category of each video frame, to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category;

[0014] A second recognition module is used to recognize a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frame contains the target object;

[0015] The processing module is used to combine multiple target frames in sequence according to the time sequence of each target frame in the video to be processed to obtain the target video.

[0016] A third aspect of an embodiment of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0017] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0018] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: in the embodiments of the present invention, a video to be processed is obtained; the scene category of each video frame in the video to be processed is determined; according to the scene category of each video frame, target recognition is performed on each video frame in the video to be processed by a trained first deep learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category; according to the target recognition result, multiple target frames of the video to be processed are identified, wherein the target frame contains the target object; according to the time sequence of each target frame in the video to be processed, the multiple target frames are combined in sequence to obtain the target video. The embodiment of the present invention determines the scene category of each video frame, and determines whether there is a target object corresponding to the scene category in each video frame through a trained deep learning model, so that targeted target recognition can be performed according to different scene categories, so that the target recognition result is more targeted and less disturbing; according to the target recognition result, multiple target frames of the video to be processed are identified, and the target video is obtained according to the target frame, so that the key information in the video to be processed can be accurately and efficiently extracted (such as identifying and extracting the image part containing the target individual), and the target video composed of the key information is obtained, while the redundant and unimportant video content in the video to be processed is deleted, and the valuable video content is retained, so that in application scenarios such as searching for key information of surveillance videos, users can perform more efficient and targeted processing according to the target video, without manually searching, editing and resynthesizing the key information. The embodiment of the present invention greatly improves the efficiency of users in obtaining key information in videos, and has strong practicality and ease of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 1 is a schematic diagram of the implementation flow of the video processing method provided in the first embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of the implementation flow of the video processing method provided in the second embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of a video processing device provided by Embodiment 3 of the present invention;

[0023] Figure 4It is a schematic diagram of a terminal device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0025] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0026] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0029] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0030] Figure 1 FIG. 1 is a schematic diagram of the implementation flow of the video processing method provided in the first embodiment of the present invention. Figure 1 The video processing method shown may include the following steps:

[0031] Step S101, obtaining a video to be processed.

[0032] The video to be processed may be a file that has undergone audio and video encoding, the encoding method of the video to be processed is known, and the encoding method is the encoding method included in the training data, wherein the training data is used to train the deep learning model.

[0033] Of course, the video to be processed may also be a video that has not been encoded, in which each frame of the image contains complete image pixel information.

[0034] Step S102: determining the scene category of each video frame in the video to be processed.

[0035] In an embodiment of the present invention, the scene category of the video frame can be determined in a variety of ways. For example, the scene category of the video frame can be determined by a preset deep learning model. It should be noted that the preset deep learning model can be the same as the first deep learning model after training described in step S103, or it can be different; the preset deep learning model can detect and determine the scene category of each video frame in the video to be processed one by one, or it can determine the subject scene category of the video to be processed based on the scene category of the video frame detected by one or more frames (for example, if the scene category of video frames greater than a preset ratio is all scene category A, then the subject scene category of the video to be processed is determined to be scene category A), and the subject scene category is used as the scene category of each video frame of the video to be processed. In addition, the scene category of each video frame in the video to be processed can also be determined based on the name, theme or preset tag of the video to be processed.

[0036] Step S103, according to the scene category of each video frame, target recognition is performed on each video frame in the video to be processed through the trained first deep learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category.

[0037] In the embodiment of the present invention, different scene categories may correspond to different target objects. For example, if the scene category of the video frame is determined to be an airport scene, the target objects corresponding to the airport scene may include airplanes, people, other mechanical equipment (such as small transportation devices in the airport), other flying objects, etc.; if the scene category of the video frame is determined to be a zoo scene, the target objects corresponding to the zoo scene may include animals, people, etc. The correspondence between the scene category and the target object may be preset.

[0038] Exemplarily, the first deep learning model may include a ResNet model, an R-CNN model, a Fast R-CNN model, etc. There may be multiple first deep learning models, which are not limited here.

[0039] Optionally, the target recognition result may include information such as the type of the target object contained in the video frame, the position of the target object in the video frame, etc.

[0040] Optionally, before performing target recognition on each video frame in the video to be processed by using the trained first deep learning model, it also includes training the first deep learning model.

[0041] Specifically, the step of training the first deep learning model may include:

[0042] Acquire training data, where the training data includes at least one training video and identification information corresponding to the training video, where the identification information may include accurate target object information (such as the type and location of the target object) in the training video;

[0043] Detecting the training data using the first deep learning model and obtaining a detection result;

[0044] Adjust the parameters of the first deep learning model according to the detection result until the detection result obtained by the adjusted first deep learning model meets the preset conditions (such as the value of the corresponding loss function is less than a preset threshold, etc.), and use the adjusted first deep learning model as the trained first deep learning model.

[0045] Step S104: identifying a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frame contains the target object.

[0046] The target frame can indicate key information in the video to be processed, and the specific identification method of the target frame of the video to be processed can be set according to the needs of the actual application scenario. For example, any frame of the video frame containing the target object can be determined as the target frame. In addition, according to the target object information, the differences between the video frames containing the target object can be judged in chronological order (the differences can include the differences of the target object, while the differences of non-target objects can be ignored), and the target frame of the video to be processed can be determined according to the differences. Exemplarily, the differences between the video frames containing the target object can be identified by a more traditional method. For example, for each video frame, if the positioning of the target object in the video frame, the size of the image area where the target object is located, etc., changes relative to the previous video frame or the next video frame; or, if the positioning of the target object in a certain video frame, the size of the area of ​​the image area, etc., relative to the previous video frame or the next video frame, the degree of difference meets the specified preset conditions (such as the positioning offset exceeds the preset offset threshold, the difference in the area of ​​the image area in two video frames accounts for the ratio of the area of ​​one of the image areas exceeds the preset ratio threshold, etc.), then the video frame is determined to be the target frame until all the video frames in the video to be processed are traversed. Of course, it is also possible to determine whether each video frame is a target frame by a preset classifier. Exemplarily, the classifier may include decision trees, logistic regression, naive Bayes, neural networks and other algorithms.

[0047] A specific example is used below to illustrate an implementation method of step S104.

[0048] For example, if it is determined that the scene category of each frame of the video to be processed is an airport scene, and the target object corresponding to the airport scene is an airplane, assuming that the video to be processed includes 20 frames of video, of which the first 10 frames of video do not include an airplane, then the first 10 frames of video are determined to be non-target frames; and in the video frames from the 11th to the 20th frames that contain an airplane, it is detected that the positions of the airplanes in the 15th to the 20th frames are offset relative to the positions of the airplanes in the previous frames of the video, then the 14th to the 20th frames of video are determined to be the target frames of the video to be processed.

[0049] Step S105 , combining a plurality of target frames in sequence according to the time sequence of each target frame in the video to be processed to obtain the target video.

[0050] In the embodiment of the present invention, the target video may only contain the target frame in the video to be processed. Therefore, it can be considered that the target video deletes the redundant and unimportant video content in the video to be processed and retains the valuable video content. Therefore, in application scenarios such as searching for key information of surveillance videos, users can perform more efficient and targeted processing based on the target video without manually searching, editing and resynthesizing the key information. In the target video, the time node information corresponding to the target frame in the video to be processed may be retained or not.

[0051] In addition, optionally, after obtaining the target video, the following steps may also be included:

[0052] The target video is stored in a specified storage space.

[0053] In the embodiment of the present invention, the storage space may be pre-set, or after obtaining the target video, the designated storage space may be determined according to the user's instruction operation, and then the target video may be stored in the designated storage space.

[0054] Since the video to be processed often contains a large number of redundant or unimportant parts, which will take up a lot of space and resources, even through existing file compression and other methods, it is not possible to improve the problem of resource occupation and waste. Therefore, in the prior art, directly storing the video to be processed will waste a lot of storage resources. In the embodiment of the present invention, the target video can be stored, so that the space required for storing the video is greatly reduced, thereby improving the problem of excessive burden on storage resources caused by the video being too large.

[0055] Optionally, if the video frame of the video to be processed contains a target object, the target recognition result also indicates the area and / or feature point position where the target object is located;

[0056] Correspondingly, the step of identifying a plurality of target frames of the video to be processed according to the target recognition result includes:

[0057] Determining whether each video frame in the video to be processed meets a first preset condition according to the area and / or feature point position of the target object indicated by the target recognition result;

[0058] The video frame meeting the first preset condition is determined as the target frame of the video to be processed.

[0059] In an embodiment of the present invention, the area where the target object is located and / or the position of the feature point may include at least one of the following information: the area of ​​the image area where the target object is located, the coordinates of the upper left corner, the coordinates of the lower right corner, the coordinates of the center point, and the coordinates of the feature points of the target object. The feature points may be determined according to the target object and the application scenario. For example, if the target object is a face, the feature points may include the feature points of the facial features. The first preset condition may be preset by a user, and there may be multiple first preset conditions. For example, it may be if the feature point coordinates of the target object in the video frame, the area of ​​the image area where the target object is located, etc., change relative to the previous video frame or the next video frame; or if the difference between the feature point coordinates of the target object in a certain video frame, the area of ​​the image area, etc., relative to the previous video frame or the next video frame meets the specified preset condition (such as the offset of the positioning exceeds the preset offset threshold, the difference in the area of ​​the image area in two video frames accounts for the ratio of the area of ​​one of the image areas exceeds the preset ratio threshold, etc.).

[0060] Optionally, judging whether each video frame in the video to be processed meets a first preset condition according to the position of the target object indicated by the target recognition result includes:

[0061] For each video frame containing the target object in the video to be processed, judging, according to the area and / or feature point position of the target object indicated by the target recognition result, whether the degree of difference between the area and / or feature point position of the target object in the video frame and that in the previous video frame or the next video frame meets a first preset condition;

[0062] Correspondingly, determining the video frame that meets the first preset condition as the target frame of the video to be processed includes:

[0063] If in the video frame, the difference in the area and / or feature point position of the target object relative to the target object in the previous video frame or the next video frame of the video frame meets the first preset condition, then the video frame is used as the target frame of the video to be processed.

[0064] Exemplarily, in an embodiment of the present invention, the degree of difference in the area where the target object is located and / or the position of the feature point may include one or more of the degree of feature point position offset, the degree of image area change, etc., wherein the degree of image area change may include the degree of change in the area of ​​the image area, or may include the ratio of the portion of the image area where the pixel value changes. At this time, exemplarily, the first preset condition may be such as the offset of the feature point coordinates of the target object exceeds a preset offset threshold, the difference in the area of ​​the image area in two video frames accounts for a ratio of the area of ​​one of the image areas that exceeds a preset ratio threshold, etc. The first preset condition may be set according to the specific application scenario and is not limited here.

[0065] Optionally, determining the scene category of each video frame in the video to be processed includes:

[0066] Performing scene recognition on each video frame in the video to be processed by using the trained second deep learning model to determine the scene category of each video frame in the video to be processed;

[0067] or,

[0068] The scene category of each video frame in the video to be processed is determined according to the name, subject or preset label of the video to be processed.

[0069] In an embodiment of the present invention, the second deep learning model may be the same as or different from the first deep learning model. Exemplarily, if the second deep learning model is the same as the first deep learning model, the second deep learning model (or the first deep learning model) may include two cascaded deep learning sub-models, and the scene category of each video frame in the video to be processed is determined by the cascaded deep learning sub-models and the target recognition is performed on each video frame in the video to be processed. Assuming that the deep learning sub-models are the first deep learning sub-model and the second deep learning sub-model, the video to be processed can be input into the first deep learning sub-model to obtain the first output result of the first deep learning sub-model, and the output result indicates the scene recognition of each video frame in the video to be processed, and then the first output result of the first deep learning sub-model is input into the second deep learning sub-model to obtain the second output result of the second deep learning sub-model, and the target recognition result can be obtained according to the second output result. Alternatively, the second deep learning model may also include only one convolutional neural network model, and the scene category and target recognition result of each video frame are obtained by the convolutional neural network model.

[0070] Exemplarily, the second deep learning model may include a ResNet model, an R-CNN model, a Fast R-CNN model, etc. There may be multiple types of the second deep learning model, which is not limited here.

[0071] Optionally, after identifying a plurality of target frames of the video to be processed, the method further includes:

[0072] The target frame is identified in the video to be processed.

[0073] Exemplarily, one or more of the time node information, scene category information, and target object information (such as the type, feature points, location, image area, etc. of the target object) corresponding to the target frame can be identified in the video to be processed. There are many ways to identify the target object. For example, the image area of ​​the target object can be identified by a rectangular frame of a specified form, the time node corresponding to the target frame can be identified in the time progress bar of the video to be processed, and so on.

[0074] Optionally, when multiple consecutive video frames of the video to be processed are all target frames, the first frame and the last frame of the multiple consecutive video frames may be identified.

[0075] The embodiment of the present invention determines the scene category of each video frame, and determines whether there is a target object corresponding to the scene category in each video frame through a trained deep learning model, so that targeted target recognition can be performed according to different scene categories, so that the target recognition result is more targeted and less disturbing; according to the target recognition result, multiple target frames of the video to be processed are identified, and the target video is obtained according to the target frame, so that the key information in the video to be processed can be accurately and efficiently extracted (such as identifying and extracting the image part containing the target individual), and the target video composed of the key information is obtained, while the redundant and unimportant video content in the video to be processed is deleted, and the valuable video content is retained, so that in application scenarios such as searching for key information of surveillance videos, users can perform more efficient and targeted processing according to the target video, without manually searching, editing and resynthesizing the key information. The embodiment of the present invention greatly improves the efficiency of users in obtaining key information of videos, and has strong practicality and ease of use.

[0076] Based on the above embodiments, Figure 2 is a schematic diagram of the implementation flow of the video processing method provided in the second embodiment of the present invention, such as Figure 2 The video processing method shown may include the following steps:

[0077] Step S201, obtaining a video to be processed;

[0078] Step S202, determining the scene category of each video frame in the video to be processed;

[0079] Step S203, according to the scene category of each video frame, performing target recognition on each video frame in the to-be-processed video by using the trained first deep learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category;

[0080] Step S204, identifying a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frame contains the target object;

[0081] Step S205, combining the plurality of target frames in sequence according to the time sequence of each target frame in the video to be processed to obtain the target video;

[0082] Steps S201, S202, S203, S204, and S205 of this embodiment are the same as steps S101, S102, S103, S104, and S105 described above. For details, please refer to the relevant descriptions of steps S101, S102, S103, S104, and S105, which will not be repeated here.

[0083] Step S206: establishing an information index according to the target frame.

[0084] In an embodiment of the present invention, the information index may include a list of links or pointers pointing to the target frame and / or to information related to the target frame. The information index may provide a pointer or link pointing to content such as specified data or nodes, so that information related to the target frame may be quickly accessed through the information index. The information index may be set in an interface associated with the video to be processed and / or associated with the target video, and the information index may point to the video to be processed and / or the target video. In addition, it should be noted that the information index may be embedded in the video to be processed or the target video, or may be an information interface (such as a database, table, text, etc.) outside the video to be processed and the target video, and there may be multiple ways to display it.

[0085] Optionally, establishing an information index according to the target frame includes:

[0086] An information index list about the target frame is established, wherein the information index list includes at least one of time information of the target frame in the video to be processed, scene category information of the target frame, and target object information of the target frame.

[0087] In an embodiment of the present invention, the time information may indicate the time node corresponding to the target frame in the video to be processed; through the time information, the time position of the target frame in the time axis of the video to be processed can be understood, so that the user can understand the situation of the target frame in different time reference systems through the time information, without losing the original time node information after the target frame is extracted from the video to be processed. The scene category information may indicate the scene category of the target frame, and the target object information may include the type, location, image area and other information related to the target object.

[0088] In the embodiment of the present invention, by establishing an information index based on the target frame, efficient retrieval of relevant information of the target frame can be achieved, which greatly improves the processing efficiency of the user's operations such as searching, editing and analyzing key information in the video to be processed.

[0089] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0090] Figure 3 FIG. 1 is a schematic diagram of a video processing device provided by Embodiment 3 of the present invention. For the convenience of explanation, only the part related to the embodiment of the present invention is shown.

[0091] The video processing device 300 comprises:

[0092] An acquisition module 301 is used to acquire a video to be processed;

[0093] A determination module 302 is used to determine the scene category of each video frame in the video to be processed;

[0094] A first recognition module 303 is used to perform target recognition on each video frame in the to-be-processed video by using a trained first deep learning model according to the scene category of each video frame, and obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category;

[0095] A second recognition module 304 is used to recognize a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frame contains the target object;

[0096] The processing module 305 is used to combine multiple target frames in sequence according to the time sequence of each target frame in the video to be processed to obtain the target video.

[0097] Optionally, if the video frame of the video to be processed contains a target object, the target recognition result also indicates the area and / or feature point position where the target object is located;

[0098] Accordingly, the second identification module 304 specifically includes:

[0099] A judging unit, configured to judge whether each video frame in the to-be-processed video meets a first preset condition according to the area and / or feature point position of the target object indicated by the target recognition result;

[0100] A determination unit is used to determine the video frame that meets the first preset condition as the target frame of the video to be processed.

[0101] Optionally, the judging unit is specifically configured to:

[0102] For each video frame containing the target object in the video to be processed, judging, according to the area and / or feature point position of the target object indicated by the target recognition result, whether the degree of difference between the area and / or feature point position of the target object in the video frame and that in the previous video frame or the next video frame meets a first preset condition;

[0103] Accordingly, the determining unit is specifically used for:

[0104] If in the video frame, the difference in the area and / or feature point position of the target object relative to the target object in the previous video frame or the next video frame of the video frame meets the first preset condition, then the video frame is used as the target frame of the video to be processed.

[0105] Optionally, the determining module 302 is specifically configured to:

[0106] Performing scene recognition on each video frame in the video to be processed by using the trained second deep learning model to determine the scene category of each video frame in the video to be processed;

[0107] or,

[0108] The scene category of each video frame in the video to be processed is determined according to the name, subject or preset label of the video to be processed.

[0109] Optionally, the video processing device 300 further includes:

[0110] The identification module is used to identify the target frame in the video to be processed.

[0111] Optionally, the video processing device 300 further includes:

[0112] The index module is used to establish an information index according to the target frame.

[0113] Optionally, the index module is specifically used for:

[0114] An information index list about the target frame is established, wherein the information index list includes at least one of time information of the target frame in the video to be processed, scene category information of the target frame, and target object information of the target frame.

[0115] The embodiment of the present invention determines the scene category of each video frame, and determines whether there is a target object corresponding to the scene category in each video frame through a trained deep learning model, so that targeted target recognition can be performed according to different scene categories, so that the target recognition result is more targeted and less disturbing; according to the target recognition result, multiple target frames of the video to be processed are identified, and the target video is obtained according to the target frame, so that the key information in the video to be processed can be accurately and efficiently extracted (such as identifying and extracting the image part containing the target individual), and the target video composed of the key information is obtained, while the redundant and unimportant video content in the video to be processed is deleted, and the valuable video content is retained, so that in application scenarios such as searching for key information of surveillance videos, users can perform more efficient and targeted processing according to the target video, without manually searching, editing and resynthesizing the key information. The embodiment of the present invention greatly improves the efficiency of users in obtaining key information in videos, and has strong practicality and ease of use.

[0116] Figure 4 Schematic diagram of a terminal device provided by Embodiment 4 of the present invention. Figure 4 As shown, the terminal device 4 of this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, the steps in the above-mentioned video processing method embodiments are implemented, for example Figure 1 Alternatively, when the processor 40 executes the computer program 42, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 3 Functions of modules 301 to 305 are shown.

[0117] Exemplarily, the computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 42 in the terminal device 4. For example, the computer program 42 may be divided into an acquisition module, a determination module, a first identification module, a second identification module, and a processing module, and the specific functions of each module are as follows:

[0118] An acquisition module is used to acquire the video to be processed;

[0119] A determination module, used to determine the scene category of each video frame in the video to be processed;

[0120] A first recognition module is used to perform target recognition on each video frame in the to-be-processed video by using a trained first deep learning model according to the scene category of each video frame, to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category;

[0121] A second recognition module is used to recognize a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frame contains the target object;

[0122] The processing module is used to combine multiple target frames in sequence according to the time sequence of each target frame in the video to be processed to obtain the target video.

[0123] The terminal device 4 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will appreciate that Figure 4 It is only an example of the terminal device 4 and does not constitute a limitation on the terminal device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0124] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0125] The memory 41 may be an internal storage unit of the terminal device 4, such as a hard disk or memory of the terminal device 4. The memory 41 may also be an external storage device of the terminal device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the terminal device 4. The memory 41 is used to store the computer program and other programs and data required by the terminal device. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0126] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0127] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0128] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0129] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. . Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0133] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A video processing method, characterized in that: include: Get the video to be processed; Determining a scene category of each video frame in the video to be processed; According to the scene category of each video frame, target recognition is performed on each video frame in the to-be-processed video by using the trained first deep learning model to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category; According to the target recognition result, identifying a plurality of target frames of the video to be processed, wherein the target frame contains the target object; Combining a plurality of target frames in sequence according to the time sequence of each target frame in the video to be processed to obtain a target video; Wherein, if the video frame of the video to be processed contains a target object, the target recognition result also indicates the area and / or feature point position where the target object is located; Correspondingly, the step of identifying a plurality of target frames of the video to be processed according to the target recognition result includes: For each video frame containing the target object in the video to be processed, judging, according to the area and / or feature point position of the target object indicated by the target recognition result, whether the degree of difference between the area and / or feature point position of the target object in the video frame and that in the previous video frame or the next video frame meets a first preset condition; If in the video frame, the degree of difference in the area and / or feature point position of the target object relative to the target object in the previous video frame or the next video frame of the video frame meets a first preset condition, then the video frame is used as the target frame of the video to be processed; the first preset condition is that the offset of the feature point coordinates of the target object exceeds a preset offset threshold, or the ratio of the difference in the area of ​​the image areas in the two video frames to the area of ​​one of the image areas exceeds a preset ratio threshold.

2. The video processing method according to claim 1, characterized in that: The determining of the scene category of each video frame in the video to be processed includes: Performing scene recognition on each video frame in the video to be processed by using the trained second deep learning model to determine the scene category of each video frame in the video to be processed; or, The scene category of each video frame in the video to be processed is determined according to the name, subject or preset label of the video to be processed.

3. The video processing method according to claim 1, characterized in that: After identifying a plurality of target frames of the video to be processed, the method further includes: The target frame is identified in the video to be processed.

4. The video processing method according to any one of claims 1 to 3, characterized in that: After identifying a plurality of target frames of the video to be processed, the method further includes: An information index is established according to the target frame.

5. The video processing method according to claim 4, characterized in that: The step of establishing an information index according to the target frame includes: An information index list about the target frame is established, wherein the information index list includes at least one of time information of the target frame in the video to be processed, scene category information of the target frame, and target object information of the target frame.

6. A video processing device, characterized in that: include: An acquisition module is used to acquire the video to be processed; A determination module, used to determine the scene category of each video frame in the video to be processed; A first recognition module is used to perform target recognition on each video frame in the to-be-processed video by using a trained first deep learning model according to the scene category of each video frame, to obtain a target recognition result, wherein the target recognition result indicates whether each video frame contains a target object, and the target object corresponds to the scene category; A second recognition module is used to recognize a plurality of target frames of the video to be processed according to the target recognition result, wherein the target frame contains the target object; A processing module, used for combining a plurality of target frames in sequence according to the time sequence of each target frame in the video to be processed to obtain a target video; Wherein, if the video frame of the video to be processed contains a target object, the target recognition result also indicates the area and / or feature point position where the target object is located; Accordingly, the second identification module specifically includes: A judging unit, configured to judge whether each video frame in the to-be-processed video meets a first preset condition according to the area and / or feature point position of the target object indicated by the target recognition result; A determination unit, configured to determine a video frame meeting the first preset condition as a target frame of the video to be processed; The judging unit is specifically used for: For each video frame containing the target object in the video to be processed, judging, according to the area and / or feature point position of the target object indicated by the target recognition result, whether the degree of difference between the area and / or feature point position of the target object in the video frame and that in the previous video frame or the next video frame meets a first preset condition; The determining unit is specifically used for: If in the video frame, the degree of difference in the area and / or feature point position of the target object relative to the target object in the previous video frame or the next video frame of the video frame meets a first preset condition, then the video frame is used as the target frame of the video to be processed; the first preset condition is that the offset of the feature point coordinates of the target object exceeds a preset offset threshold, or the ratio of the difference in the area of ​​the image areas in the two video frames to the area of ​​one of the image areas exceeds a preset ratio threshold.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the video processing method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 5 are implemented.

Images