Video detection method, video detection device, storage medium and electronic device

Abstract

The present disclosure provides a video detection method, a video detection device, a computer readable storage medium and an electronic device, and relates to the technical field of video processing. The video detection method comprises: acquiring a to-be-detected video and a reference video of a target action; extracting features of image frames in the to-be-detected video to obtain a to-be-detected feature sequence corresponding to the to-be-detected video, and extracting features of image frames in the reference video to obtain a reference feature sequence corresponding to the reference video; determining a target sub-feature sequence matched with the reference feature sequence in the to-be-detected feature sequence; and determining image frames related to the target action in the to-be-detected video according to the target sub-feature sequence. The present disclosure can conveniently and quickly detect image frames related to the target action in the to-be-detected video.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video processing, and particularly relates to a video detection method, a video detection device, a computer readable storage medium and an electronic device. BACKGROUND

[0002] With the rapid development of computer technology, image and video data are growing rapidly. In order to meet the diversified needs of users in various video processing scenarios, it is often necessary to detect videos. For example, when a user wants to extract a sub-video of a jumping action from a long video, the long video can be detected to identify the jumping action and extract the sub-video corresponding to the jumping action or determine the start time and end time of the jumping action.

[0003] The prior art video detection method usually needs to pre-train a deep neural network model based on a large amount of labeled data, and processes the video to be detected by the deep neural network model to realize the recognition of the specified action. However, in order to ensure the accuracy of video detection, when using this method, a large amount of training videos need to be labeled manually, which has high time and labor costs. Moreover, as the demand for video detection increases, the size of the training video set is also limited, which affects the performance of the deep neural network model, resulting in poor video detection efficiency and accuracy. SUMMARY

[0004] The present disclosure provides a video detection method, a video detection device, a computer readable storage medium and an electronic device, thereby at least partially solving the problem that the prior art video detection method needs to label a large amount of training videos, which has high time and labor costs.

[0005] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0006] According to a first aspect of the present disclosure, a video detection method is provided, comprising: obtaining a to-be-detected video and a reference video of a target action; extracting features of image frames in the to-be-detected video to obtain a to-be-detected feature sequence corresponding to the to-be-detected video, and extracting features of image frames in the reference video to obtain a reference feature sequence corresponding to the reference video; determining a target sub-feature sequence matched with the reference feature sequence in the to-be-detected feature sequence; and determining image frames related to the target action in the to-be-detected video according to the target sub-feature sequence.

[0007] According to a second aspect of the present disclosure, a video detection apparatus is provided, comprising: a video acquisition module configured to acquire a to-be-detected video and a reference video of a target action; a feature extraction module configured to extract features of image frames in the to-be-detected video to obtain a to-be-detected feature sequence corresponding to the to-be-detected video, and extract features of image frames in the reference video to obtain a reference feature sequence corresponding to the reference video; a sequence determination module configured to determine a target sub-feature sequence matching the reference feature sequence in the to-be-detected feature sequence; and an image determination module configured to determine image frames related to the target action in the to-be-detected video according to the target sub-feature sequence.

[0008] According to a third aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the video detection method of the first aspect and possible implementation manners thereof.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory configured to store executable instructions of the processor. The processor is configured to execute the executable instructions to implement the video detection method of the first aspect and possible implementation manners thereof.

[0010] The technical solution of the present disclosure has the following beneficial effects:

[0011] The to-be-detected video and the reference video of the target action are acquired. Features of image frames in the to-be-detected video are extracted to obtain a to-be-detected feature sequence corresponding to the to-be-detected video, and features of image frames in the reference video are extracted to obtain a reference feature sequence corresponding to the reference video. A target sub-feature sequence matching the reference feature sequence in the to-be-detected feature sequence is determined. Image frames related to the target action in the to-be-detected video are determined according to the target sub-feature sequence. On the one hand, the present exemplary embodiment proposes a new video detection method. By comparing the to-be-detected feature sequence and the reference feature sequence, image frames related to the target action in the to-be-detected video are determined. Since the target sub-feature sequence is matched based on the reference video of the target action, the matching process is based on the reference video, so that the detection result has strong pertinence and accuracy. On the other hand, when performing matching, only the processing process of the reference video of the target action is involved, without the need to process a large number of training videos. Compared with video detection by using a deep neural network model with high training complexity by labeling a large amount of data, the time cost and labor cost of video data processing are greatly reduced, and the video detection process is simpler and more convenient.

[0012] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate implementations of the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings, described below, are only some embodiments of the present disclosure and that other drawings can be obtained by those of ordinary skill in the art without creative effort based on these drawings.

[0014] Figure 1 A schematic diagram showing a system architecture in the present exemplary embodiment;

[0015] Figure 2 A structural diagram showing an electronic device in the present exemplary embodiment;

[0016] Figure 3 A flowchart showing a video detection method in the present exemplary embodiment;

[0017] Figure 4 An image frame schematic diagram showing a video detection method in the present exemplary embodiment;

[0018] Figure 5 A sub-flowchart showing a video detection method in the present exemplary embodiment;

[0019] Figure 6 A schematic diagram showing training of a feature extraction model in the present exemplary embodiment;

[0020] Figure 7 A flowchart showing another video detection method in the present exemplary embodiment;

[0021] Figure 8 Another sub-flowchart showing a video detection method in the present exemplary embodiment;

[0022] Figure 9 A structural block diagram showing a video detection apparatus in the present exemplary embodiment. DETAILED DESCRIPTION

[0023] Example implementations are now described with reference to the following drawings. The example implementations, can, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example implementations to those skilled in the art. The features, structures, or characteristics described can be combined in one or more implementations. In the following description, numerous specific details are provided, such as examples of programming, software modules, user terminals, networks, etc., to provide a thorough understanding of the example implementations. One skilled in the relevant art will recognize, however, that the implementations can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. Some features, structures, or characteristics can be described as being part of one or more implementations, but can be combined in other implementations. In other instances, well-known features, structures or characteristics are not described in detail to avoid obscuring the related descriptions in the one or more implementations.

[0024] Furthermore, the accompanying drawings are only schematic and are non-limiting examples, as the disclosure can be implemented in various forms. Identical, corresponding, or similar elements present in multiple figures are referred to by the same reference numerals, and only the differing features from the previous figures are described in detail. Some features, structures, or characteristics can be shown in highly diagrammatic form, exaggerated in scale, or used as a symbol for a greater group, in order to avoid obscuring the relevant parts of the drawings.

[0025] An example implementation of the present disclosure provides a video detection method. Figure 1 A system architecture diagram of an operating environment of the example implementation is shown. As shown in the figure, the system architecture 100 can include a user terminal 110 and a server 120, which can form a communication interaction through a network, for example, the user terminal 110 can send the obtained video data to the server 120, and the server 120 can return the video detection result to the user terminal 110. Among them, the user terminal 110 can include but is not limited to smart phones, tablet computers, game consoles, wearable devices, and other electronic devices; the server 120 refers to a background server that provides Internet services or video processing capabilities. Figure 1

[0026] It should be understood that Figure 1 The number of devices in the figure is only exemplary. According to the needs of implementation, any number of user terminals can be set, or the server can be a cluster formed by multiple servers.

[0027] ​The video detection method provided by the embodiments of the present disclosure can be executed by the user terminal 110, for example, the user terminal 110 directly executes the video detection process after collecting the video; or the video detection method can be executed by the server 120, for example, the user terminal 110 uploads the collected video to the server 120, and the server 120 executes the video detection process and returns the detection result to the user terminal 110, and the like, and the present disclosure does not limit the same.

[0028] The exemplary embodiments of the present disclosure provide an electronic device for implementing the video detection method, which can be the user terminal 110 or the server 120 in Figure 1 The electronic device at least includes a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the video detection method by executing the executable instructions.

[0029] The following takes the mobile terminal 200 in Figure 2 as an example to exemplarily illustrate the structure of the above-mentioned electronic device. Those skilled in the art should understand that, in addition to the components specially used for mobile purposes, Figure 2 The structure in

[0030] As shown in Figure 2 , the mobile terminal 200 can specifically include a processor 210, an internal memory 221, an external memory interface 222, a USB (Universal Serial Bus, Universal Serial Bus) interface 230, a charge management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a loudspeaker 271, a receiver 272, a microphone 273, a headset interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a key 294, and a SIM (Subscriber Identification Module, Subscriber Identification Module) card interface 295, and the like.

[0031] The processor 210 can include one or more processing units, for example: the processor 210 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0032] The encoder can encode (i.e., compress) image or video data, for example, encode the to-be-detected video or the reference video obtained after video quality processing, to form corresponding code stream data, so as to reduce the bandwidth occupied by data transmission; the decoder can decode (i.e., decompress) the code stream data of the image or video, to restore the image or video data, for example, decode the to-be-detected video or the reference video, to obtain each frame of image data in the video, and perform feature extraction on one or more frames of images, etc.

[0033] In some embodiments, the processor 210 can include one or more interfaces, through which the different interfaces and other components of the mobile terminal 200 are connected.

[0034] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 221 can include volatile memory, non-volatile memory, etc. The processor 210 performs various functional applications and data processing of the mobile terminal 200 by running the instructions stored in the internal memory 221 and / or the instructions stored in the memory arranged in the processor.

[0035] The external memory interface 222 can be used to connect an external memory, for example, a Micro SD card, to realize the expansion of the storage capacity of the mobile terminal 200. The external memory communicates with the processor 210 through the external memory interface 222, to realize the data storage function, for example, storing music, video, etc.

[0036] The USB interface 230 is an interface conforming to the USB standard specification, which can be used to connect a charger to charge the mobile terminal 200, or connect a headset or other electronic devices.

[0037] The charging management module 240 is used to receive charging input from the charger. The charging management module 240 can supply power to the device while charging the battery 242; the power management module 241 can also monitor the state of the battery.

[0038] The wireless communication function of the mobile terminal 200 can be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, and the baseband processor, etc. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 250 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the mobile terminal 200. The wireless communication module 260 can provide a wireless communication solution including WLAN (Wireless Local Area Networks) (such as Wi-Fi (Wireless Fidelity) network), BT (Bluetooth), GNSS (Global Navigation Satellite System), FM (Frequency Modulation), NFC (Near Field Communication), IR (Infrared), etc. applied to the mobile terminal 200.

[0039] The mobile terminal 200 can realize the display function through the GPU, the display screen 290, and the AP, etc., and display the user interface. The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the encoder, the decoder, the GPU, the display screen 290, and the AP, etc., and can also realize the audio function through the audio module 270, the speaker 271, the receiver 272, the microphone 273, the earphone interface 274, and the AP, etc.

[0040] The sensor module 280 can include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, a barometric pressure sensor 2804, etc., to realize different sensing and detecting functions.

[0041] The indicator 292 can be an indicator light, which can be used to indicate the charging state, the power change, and can also be used to indicate messages, missed calls, notifications, etc. The motor 293 can generate a vibration prompt, and can also be used for touch vibration feedback, etc. The key 294 includes a power-on key, a volume key, etc.

[0042] The mobile terminal 200 can support one or more SIM card interfaces 295 for connecting SIM cards to realize the functions of calling and data communication, etc.

[0043] Figure 3 An exemplary flow of the video detection method is shown, which can be executed by the user terminal 110 or the server 120, and includes the following steps S310-S340:

[0044] Step S310, obtaining a reference video of a target action for a to-be-detected video.

[0045] The to-be-detected video refers to a video in which a target action needs to be detected. The to-be-detected video can be a video captured by a user in real time or historically, a local video, or a video downloaded from a cloud or other video sources. The target action refers to an action that needs to be detected. According to actual detection requirements, the target action can include multiple types. For example, when a running action needs to be detected in the to-be-detected video, the target action is the running action; when a jumping action needs to be detected in the to-be-detected video, the target action is the jumping action, and so on. The target action can be one action, such as a running action in the to-be-detected video. The target action can also be multiple actions, such as a running action or a jumping action in the to-be-detected video. The target action can also be a combination of multiple actions, such as a combination of a running action and a hurdling action in the to-be-detected video, and so on.

[0046] The reference video refers to an example video that includes a target action and is used to compare with the to-be-detected video to detect whether the to-be-detected video contains the target action. In this example embodiment, the reference video can be one. For example, when only a running action needs to be detected in the to-be-detected video, a video containing the running action can be obtained as the reference video. The reference video can also be multiple. For example, when a running action and a jumping action need to be detected in the to-be-detected video, a video containing the running action and a video containing the jumping action can be obtained as the reference video, and so on. Alternatively, to improve the accuracy of detecting the target action, when a running action needs to be detected in the to-be-detected video, multiple reference videos containing the running action can be obtained. For example, videos of different people running or videos of different scenes of running can be obtained, and the present disclosure does not make a specific limitation in this regard.

[0047] Step S320, extracting features of image frames in the to-be-detected video to obtain a to-be-detected feature sequence corresponding to the to-be-detected video, and extracting features of image frames in the reference video to obtain a reference feature sequence corresponding to the reference video.

[0048] The example embodiment can first extract a feature from an image frame in the to-be-detected video, where the feature refers to data capable of reflecting information of the image frame, and is used to represent the corresponding image frame. The feature can be extracted from the image frame by encoding the image frame, and the feature can be a feature vector generated by encoding. Then, a to-be-detected feature sequence corresponding to the to-be-detected video can be generated according to the time sequence of the image frame in the to-be-detected video. When the feature of the image frame is a feature vector, the corresponding to-be-detected feature sequence is a feature vector sequence. The example embodiment can input the to-be-detected video into a specific encoder or a pre-trained network model to encode the image frame, extract the feature of the image frame, and further generate the to-be-detected feature sequence.

[0049] In the example embodiment, when extracting the feature of the image frame in the to-be-detected video, the feature of all image frames in the to-be-detected video can be extracted, for example, the to-be-detected video is input into an encoder to encode all image frames in the to-be-detected video frame by frame to generate a feature vector of each frame. The feature of part of the image frames in the to-be-detected video can also be extracted, for example, when detecting a target action of a to-be-detected video shot by a user, the target action often appears in the middle or middle-late section, so the feature of the image frames other than the image frames of the head part of the preset number of frames in the to-be-detected video can be extracted. The feature of the image frames in the to-be-detected video can also be extracted according to the user's demand, for example, the user selects to detect a certain section of the to-be-detected video, and the feature of the image frames in the section of the video can be extracted, and so on. The image frames for which the feature is extracted can be determined according to actual conditions, and the disclosure does not make specific limitations thereto.

[0050] In addition to extracting the feature from the image frame in the to-be-detected video to obtain the to-be-detected feature sequence corresponding to the to-be-detected video, the example embodiment can also extract the feature of the image frame in the obtained reference video to obtain the reference feature sequence corresponding to the reference video. The specific feature extraction method of the image frame of the reference video and the generation method of the reference feature sequence can be the same as the feature extraction method of the image frame in the to-be-detected video and the generation method of the to-be-detected feature sequence, that is, the feature vector of the image frame of the reference video can also be obtained by inputting the reference video into the pre-trained network model, and further, the reference feature sequence corresponding to the reference video can be obtained.

[0051] In step S330, a target sub-feature sequence matching the reference feature sequence is determined in the to-be-detected feature sequence.

[0052] The to-be-detected feature sequence reflects a to-be-detected video, and the reference feature sequence reflects a reference video containing a target action. The process of searching for a target sub-feature sequence matching the reference feature sequence in the to-be-detected feature sequence can be regarded as a process of determining a sub-video about the target action in the to-be-detected video. The target sub-feature sequence is a sub-sequence of image frame features in the to-be-detected feature sequence that can be related to the target action.

[0053] In the present exemplary embodiment, the target sub-feature sequence can be determined in various ways. Specifically, the features of each image frame in the reference feature sequence can be taken as a reference to search for corresponding to-be-detected features in the to-be-detected feature sequence, and then the target sub-feature sequence can be determined according to the search result. Alternatively, the continuous distribution pattern of the reference feature sequence can be taken as a reference to compare and search for a sub-feature sequence similar to the continuous distribution pattern of the reference feature sequence in the to-be-detected feature sequence. For example, the size of a sliding window can be determined according to the length of the reference feature sequence, and then the sliding window can be used to calculate the matching degree between the feature sequence in the sliding window and the reference feature sequence in the to-be-detected feature sequence, so as to determine the target sub-feature sequence matching the reference feature sequence in the to-be-detected feature sequence. The matching degree can be calculated in various ways. For example, the Euclidean distance can be used to calculate the similarity between the feature sequence in the sliding window and the reference feature sequence, and the similarity can be used to represent the matching degree of the feature sequence. When no sub-feature sequence matching the reference feature sequence is found in the to-be-detected feature sequence, it indicates that there is no sub-video about the target action in the to-be-detected video.

[0054] In step S340, the image frame related to the target action in the to-be-detected video is determined according to the target sub-feature sequence.

[0055] The target sub-feature sequence is a sequence composed of features of image frames related to the target action in the to-be-detected feature sequence. Based on each feature in the target sub-feature sequence, the image frames included in the sub-video related to the target action can be determined from the to-be-detected video. For example, each feature vector in the target sub-feature sequence can be used to determine the image frame corresponding to the feature vector from the to-be-detected video. According to actual needs, all image frames related to the target action can be determined from the to-be-detected video. For example, when the target action is a jumping action, all image frames included in the sub-video corresponding to the start time of the jumping action to the end time of the jumping action can be determined from the to-be-detected video. In addition, part of the image frames related to the target action can also be determined from the to-be-detected video. For example, when the target action is a jumping action, the image frame corresponding to the start time of the jumping action and the image frame corresponding to the end time of the jumping action can be determined from the to-be-detected video. When the target action includes multiple actions, the image frames related to the multiple target actions can be output by the example embodiment. In order to facilitate management, the target actions can be classified, and the image frames related to each type of target action can be identified, for example, 10 image frames related to a jumping action and 15 image frames related to a running action can be output, wherein the jumping action is in the 16th frame to the 25th frame of the to-be-detected video, and the running action is in the 31st frame to the 45th frame of the to-be-detected video, and so on.

[0056] The example embodiment detects the target action in the to-be-detected video based on the comparison between the to-be-detected feature sequence and the reference feature sequence, and can maintain high detection accuracy and efficiency even when the person performing the target action is partially occluded in the to-be-detected video.

[0057] In an example embodiment, the step S340 described above can include:

[0058] According to the position of the target sub-feature sequence in the to-be-detected feature sequence, the start frame and the end frame of the target action in the to-be-detected video are determined.

[0059] In actual applications, there are often application scenarios that require detecting and determining the boundaries of the target action in the video. The example embodiment can determine the start frame and the end frame of the target action in the to-be-detected video according to the position of the target sub-feature sequence in the to-be-detected feature sequence. The start frame and the end frame are used to identify the motion boundaries of the target action in the to-be-detected video. For example, when the target action is a running action and the target sub-feature sequence of the running action is in the 16th frame to the 25th frame of the to-be-detected feature sequence, the 16th frame and the 25th frame of the to-be-detected video can be determined as the start frame and the end frame of the running action, respectively. Two key frames in the multiple image frames of the running action are used to represent the motion boundaries of the running action in the to-be-detected video, and so on.

[0060] Figure 4A schematic diagram of a video detection method in the example embodiment is shown as follows. Figure 4 As shown in the figure, taking a jumping action as an example, first, a to-be-detected video 410 and a reference video 420 containing a jumping action can be obtained; then features of image frames in the to-be-detected video 410 are extracted to obtain a to-be-detected feature sequence 430 corresponding to the to-be-detected video, and features of image frames in the reference video are extracted to obtain a reference feature sequence 440 corresponding to the reference video; a target sub-feature sequence matching the reference feature sequence 440 is determined in the to-be-detected feature sequence 430; finally, all image frames 450 related to the target action in the to-be-detected video 410 are determined according to the target sub-feature sequence, or the start frame and the end frame 460 of the target action in the to-be-detected video 410 are determined according to the position of the target sub-feature sequence in the to-be-detected feature sequence 430.

[0061] In summary, in the example embodiment, a to-be-detected video and a reference video of a target action are obtained; features of image frames in the to-be-detected video are extracted to obtain a to-be-detected feature sequence corresponding to the to-be-detected video, and features of image frames in the reference video are extracted to obtain a reference feature sequence corresponding to the reference video; a target sub-feature sequence matching the reference feature sequence is determined in the to-be-detected feature sequence; and image frames related to the target action in the to-be-detected video are determined according to the target sub-feature sequence. On the one hand, the example embodiment proposes a new video detection method, which determines image frames related to the target action in the to-be-detected video by comparing the to-be-detected feature sequence and the reference feature sequence. Since the target sub-feature sequence is matched based on the reference video of the target action, the matching process is based on the reference video, so that the detection result has strong pertinence and accuracy. On the other hand, the example embodiment only involves the processing of the reference video of the target action when matching, without the need to process a large number of training videos. Compared with video detection by a deep neural network model with high training complexity and a large amount of labeled data, the time cost and labor cost of video data processing are greatly reduced, and the video detection process is simpler and more convenient.

[0062] In an example embodiment, the step S320 of extracting features of image frames in the to-be-detected video to obtain a to-be-detected feature sequence corresponding to the to-be-detected video comprises:

[0063] The features of the image frames in the to-be-detected video are extracted by using a pre-trained feature extraction model to obtain a to-be-detected feature sequence corresponding to the to-be-detected video.

[0064] The example embodiment can pre-train a feature extraction model, and encode the to-be-detected video by using the feature extraction model to extract image features of image frames in the to-be-detected video, and then obtain a to-be-detected feature sequence corresponding to the to-be-detected video. The feature extraction model can be a self-supervised deep neural network model trained based on a temporal consistency algorithm, which does not need to be provided with explicit labels, and can be trained by inputting pairs of sample videos for encoding learning.

[0065] In addition, the feature extraction model can also be other neural network models with image feature extraction functions. For example, when a to-be-classified image is input into an image classification model, the to-be-classified image is often first processed by an intermediate layer to extract image features and generate a feature image, and then the feature image is classified and recognized to obtain an image classification result. Therefore, the image classification model can also be used as the feature extraction model in the example embodiment, and the image classification model is used to process image frames in the to-be-detected video to only obtain the feature image output by the intermediate layer, so as to implement the process of extracting features of the image frames in the to-be-detected video, and the like.

[0066] Specifically, in an example embodiment, as shown in Figure 5 The video detection method can further include the following steps:

[0067] In step S510, a sample video pair is obtained, the sample video pair including a first sample video and a second sample video, and the first sample video and the second sample video corresponding to the same action;

[0068] In step S520, a feature extraction model to be trained is used to extract features of image frames in the first sample video and the second sample video, respectively, to obtain a first sample feature sequence corresponding to the first sample video and a second sample feature sequence corresponding to the second sample video.

[0069] In step S530, for at least one first sample feature in the first sample feature sequence, a second sample feature most similar to the first sample feature is determined in the second sample feature sequence to obtain a first matching result, and a first sample feature most similar to the second sample feature is determined in the first sample feature sequence to obtain a second matching result.

[0070] In step S540, parameters of the feature extraction model are updated according to differences between the first matching result and the second matching result.

[0071] The sample video pair refers to training data used to train the feature extraction model. The sample video pair can include a first sample video and a second sample video of the same action, for example, a first sample video and a second sample video of running in different scenes, or a first sample video or a second sample video of running by different people, etc. During training of the feature extraction model, the sample video pair can be used as training data and input into the feature extraction model to be trained, for example, in batches or epochs. Then, the feature extraction model to be trained is used to extract features from the image frames in the first sample video and the second sample video, for example, to extract feature vectors e1 of the image frames in the first sample video and feature vectors e2 of the image frames in the second sample video. Then, based on the features extracted from the image frames in the first sample video and the second sample video, a first sample feature sequence corresponding to the first sample video and a second sample feature sequence corresponding to the second sample video are generated.

[0072] Further, for at least one frame of the first sample feature sequence, the most similar second sample feature in the second sample feature sequence is determined to obtain a first matching result. Then, the most similar first sample feature in the first sample feature sequence to the second sample feature is determined to obtain a second matching result. That is, the image frame similar to the image frame in the first sample video is found in the second sample video, and then the image frame most similar to the image frame is found in the first sample video. Thus, the difference between the first matching result and the second matching result, i.e., the difference between the image frame found in the second sample video and the image frame found in the first sample video, is used to update the parameters of the feature extraction model. Specifically, the parameters of the feature extraction model can be adjusted to make the difference between the first matching result and the second matching result smaller and smaller until the accuracy of the feature extraction model reaches a certain standard or a convergence condition is reached, thereby obtaining the final feature extraction model after parameter update. Based on the above method, the training process of the feature extraction model can be realized. Figure 6 A training diagram of a feature extraction model is shown, which can include: obtaining a sample video pair, the sample video pair including a first sample video 610 and a second sample video 620, the first sample video 610 and the second sample video 620 corresponding to the same action; extracting features from image frames in the first sample video 610 and the second sample video 620 by a feature extraction model 630 to be trained; constructing a time domain cycle alignment loss function 640 based on a time cycle consistency algorithm, updating parameters of the feature extraction model, and obtaining a trained feature extraction model.

[0073] Figure 7A flowchart of another video detection method is shown, which can specifically include the following steps:

[0074] In step S710, a video to be detected is acquired.

[0075] In step S720, a reference video containing a target action is acquired.

[0076] In step S730, a pre-trained feature extraction model is used to extract features of image frames in the video to be detected, to obtain a to-be-detected feature sequence corresponding to the video to be detected.

[0077] In step S740, a pre-trained feature extraction model is used to extract features of image frames in the reference video, to obtain a reference feature sequence corresponding to the reference video.

[0078] In step S750, it is determined whether there is a target sub-feature sequence matching the reference feature sequence in the to-be-detected feature sequence.

[0079] If there is a target feature sequence matching the reference feature sequence in the to-be-detected feature sequence, step S760 is performed.

[0080] In step S760, the category of the target action, and the starting frame and ending frame of the target action are determined in the video to be detected according to the position of the target sub-feature sequence in the to-be-detected feature sequence.

[0081] If there is no target feature sequence matching the reference feature sequence in the to-be-detected feature sequence, step S770 is performed.

[0082] In step S770, a detection result that there is no target feature sequence matching the reference feature sequence in the to-be-detected feature sequence is returned.

[0083] In an exemplary embodiment, as shown in Figure 8 The above step S330 can include:

[0084] In step S810, the size of the sliding window is determined according to the length of the reference feature sequence.

[0085] In step S820, a sub-feature sequence is extracted from the to-be-detected feature sequence using the sliding window, and the matching degree of the sub-feature sequence and the reference feature sequence is determined, and when the matching degree reaches a preset threshold, the sub-feature sequence is determined as the target sub-feature sequence.

[0086] In the present exemplary embodiment, the matching process of the feature sequence can be implemented by setting a sliding window and sliding in the video to be detected. Specifically, the size of the sliding window can be determined according to the length of the reference feature sequence, for example, if the reference feature sequence is a sequence composed of features of 10 frames of images, the size of the sliding window can be set to 10 frames, or can be set to less than 10 frames, such as 9 frames or 8 frames, etc. Then, the sliding window can be used to slide in the feature sequence to be detected, each step of sliding can determine a sub-feature sequence corresponding to the current sliding window, and further, the target sub-feature sequence can be determined by calculating the matching degree between the sub-feature sequence and the reference feature sequence. The step size of the sliding window can be self-defined according to the speed requirement and accuracy requirement of the balanced system, which is not limited in the present disclosure. The matching degree between the reference feature sequence and the sub-feature sequence can be realized by similarity calculation, for example, the cosine similarity or Euclidean distance can be used to calculate the similarity between the feature vectors in the reference feature sequence and the feature vectors in the sub-feature sequence, when the matching degree reaches a preset threshold, for example, the Euclidean distance is less than a preset threshold, it is considered that the current matching degree meets certain requirements, and the sub-feature sequence can be determined as the target sub-feature sequence.

[0087] In an exemplary embodiment, the step S820 of extracting the sub-feature sequence in the feature sequence to be detected by using the sliding window can include the following steps:

[0088] determining at least one reference feature frame in the reference feature sequence;

[0089] determining the most similar feature frame to be detected in the feature sequence to be detected, and determining the initial position of the sliding window in the feature sequence to be detected according to the position of the feature frame to be detected;

[0090] placing the sliding window at the initial position and extracting the sub-feature sequence located in the sliding window.

[0091] In the present exemplary embodiment, in order to improve the efficiency and accuracy of the sliding window in the feature sequence to be detected, the initial position of the sliding window can be specially set. Specifically, at least one reference feature frame can be determined in the reference feature sequence, which can be the first frame or the last frame image frame corresponding to the reference feature sequence. Then the most similar feature frame to be detected in the reference feature sequence is found, for example, the feature sequence to be detected is generated from the video containing the jumping action, and the reference feature sequence is the video containing the jumping action. The reference feature frame can be set as the image frame in which the foot of the character just separates from the ground at the beginning of the jumping action. Since the feature sequence to be detected includes video segments other than the jumping action of the character, if the sliding window starts from the head of the feature sequence to be detected, a large number of invalid matching calculations can be generated. In order to improve the detection efficiency of the sliding window in the feature sequence to be detected, the most similar feature frame to be detected to the reference feature frame can be found in the feature sequence to be detected, such as finding the image frame in which the foot of the character just separates from the ground as the feature frame to be detected. The specific searching method can be determined by calculating the similarity between the feature frames, for example, the similarity between the reference feature frame and the image frame in the feature sequence to be detected can be determined by cosine similarity or Euclidean distance, and the most similar feature frame to be detected to the reference feature frame can be determined according to the similarity. Further, the initial position of the sliding window in the feature sequence to be detected can be determined according to the position of the feature frame to be detected, for example, the position of the feature frame to be detected can be determined as the initial position of the sliding window in the feature sequence to be detected; or the position of the feature frame to be detected can be determined as the end position of the sliding window in the feature sequence to be detected, and the initial position can be determined by backtracking according to the preset step size. Finally, the sliding window starts to move from the determined initial position, and the sub-feature sequence in the sliding window is extracted after each movement, for example, the first frame in the reference feature sequence can be taken as the reference feature frame, and the most similar feature frame to be detected to the reference feature frame can be found in the feature sequence to be detected. The position of the feature frame to be detected can be taken as the initial position of the sliding window, and the sliding window can be moved according to the preset step size. After moving one step, the sub-feature sequence in the current sliding window can be extracted.

[0092] In an exemplary embodiment, the reference video of the target action can include multiple reference videos. When the target action includes multiple reference videos, the lengths of the reference videos can be inconsistent. Based on this, the size of the sliding window can be determined in various ways in the present exemplary embodiment, for example, the size of the sliding window can be determined according to the average value calculated by calculating the average value of the lengths of different reference videos. The size of the sliding window can also be determined by comparing multiple reference feature sequences corresponding to multiple reference videos. Specifically, the above step S510 can include:

[0093] compare the plurality of reference feature sequences corresponding to the plurality of reference videos to determine outlier feature frames in the reference feature sequences;

[0094] remove the outlier feature frames from the reference feature sequences, and determine the size of the sliding window based on the length of the reference feature sequences after removing the outlier feature frames.

[0095] The outlier feature frame refers to an image frame in a feature sequence that is greatly different from other feature frames or has abnormal data. In order to ensure the accuracy and effectiveness of the reference feature sequence, when the lengths of the plurality of reference videos are inconsistent, the plurality of reference feature sequences corresponding to the plurality of reference videos can be compared to determine the outlier feature frames in the reference feature sequences. Then, the outlier feature frames are removed from the reference feature sequences, and the size of the sliding window is determined based on the length of the reference feature sequences after removing the outlier feature frames, thereby ensuring the effectiveness of the remaining feature frames and further ensuring the accuracy of the feature sequence matching.

[0096] It should be noted that the size of the sliding window can be flexibly set according to actual needs or reference videos. For example, different sizes of the sliding window can be set for reference videos of different categories of target actions, and the present disclosure does not make specific limitations thereto.

[0097] The exemplary embodiments of the present disclosure also provide a video detection device. As shown in Figure 9 The video detection device 900 can include: a video acquisition module 910, configured to acquire a to-be-detected video and a reference video of a target action; a feature extraction module 920, configured to extract features of image frames in the to-be-detected video to obtain a to-be-detected feature sequence corresponding to the to-be-detected video, and extract features of image frames in the reference video to obtain a reference feature sequence corresponding to the reference video; a sequence determination module 930, configured to determine a target sub-feature sequence matched with the reference feature sequence in the to-be-detected feature sequence; and an image determination module 940, configured to determine image frames related to the target action in the to-be-detected video according to the target sub-feature sequence.

[0098] In an exemplary embodiment, the feature extraction module includes: a feature extraction unit, configured to extract features of image frames in the to-be-detected video by using a pre-trained feature extraction model to obtain the to-be-detected feature sequence corresponding to the to-be-detected video.

[0099] In an example embodiment, the video detection apparatus further comprises: a sample video obtaining module configured to obtain a sample video pair, the sample video pair comprising a first sample video and a second sample video, the first sample video and the second sample video corresponding to the same action; a sample feature extracting module configured to extract features of image frames in the first sample video and the second sample video respectively by using the feature extraction model to be trained, to obtain a first sample feature sequence corresponding to the first sample video and a second sample feature sequence corresponding to the second sample video; a matching result obtaining module configured to, for at least one first sample feature in the first sample feature sequence, determine a second sample feature in the second sample feature sequence that is most similar to the first sample feature, to obtain a first matching result, and determine a first sample feature in the first sample feature sequence that is most similar to the second sample feature, to obtain a second matching result; and a model parameter updating module configured to update parameters of the feature extraction model according to a difference between the first matching result and the second matching result.

[0100] In an example embodiment, the sequence determining module comprises: a length determining unit configured to determine a size of the sliding window according to a length of the reference feature sequence; and a target sub-feature sequence determining unit configured to extract a sub-feature sequence from the feature sequence to be detected by using the sliding window, and determine a matching degree of the sub-feature sequence and the reference feature sequence, and determine the sub-feature sequence as the target sub-feature sequence when the matching degree reaches a preset threshold.

[0101] In an example embodiment, the target sub-feature sequence determining unit comprises: a reference feature frame determining sub-unit configured to determine at least one reference feature frame in the reference feature sequence; an initial position determining sub-unit configured to determine a feature frame in the feature sequence to be detected that is most similar to the reference feature frame, and determine an initial position of the sliding window in the feature sequence to be detected according to a position of the feature frame in the feature sequence to be detected; and a sub-feature sequence extracting sub-unit configured to place the sliding window at the initial position and extract the sub-feature sequence located in the sliding window.

[0102] In an example embodiment, the reference video of the target action comprises a plurality of reference videos; and the target sub-feature sequence determining unit comprises: an outlier feature frame determining sub-unit configured to compare a plurality of reference feature sequences corresponding to the plurality of reference videos, to determine an outlier feature frame in the reference feature sequence; and an outlier feature frame eliminating sub-unit configured to eliminate the outlier feature frame from the reference feature sequence, and determine the size of the sliding window according to a length of the reference feature sequence after the outlier feature frame is eliminated.

[0103] In an example embodiment, the image determining module comprises: an image frame determining unit configured to determine a starting frame and an ending frame of the target action in the video to be detected according to a position of the target sub-feature sequence in the feature sequence to be detected.

[0104] The specific details of parts of the above-described apparatus have been described in detail in the method part embodiments, and thus will not be described again.

[0105] The exemplary embodiments of the present disclosure also provide a computer readable storage medium, which can be implemented in the form of a program product, including program codes for causing a terminal device to perform any one or more steps of the above-described "exemplary method" part of the present specification, for example, the steps according to various exemplary embodiments of the present disclosure described in the above-described "exemplary method" part of the present specification, when the program product is run on the terminal device. Figure 3 , Figure 5 Figure 7 or Figure 8 The program product can be in the form of a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in the present document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus or device.

[0106] The program product can be in any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0107] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the readable program codes are carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a readable storage medium and that can transmit, propagate or transport the program for use by or in connection with an instruction execution system, apparatus or device.

[0108] The program codes contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0109] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0110] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0111] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A video detection method, characterized by, The method comprises: obtaining a reference video of a target action and a video to be detected; extracting features of image frames in the video to be detected to obtain a feature sequence corresponding to the video to be detected, and extracting features of image frames in the reference video to obtain a reference feature sequence corresponding to the reference video; determining a target sub-feature sequence matching the reference feature sequence in the feature sequence to be detected; the target sub-feature sequence is a group sequence composed of features of image frames related to the target action in the feature sequence to be detected; determining image frames related to the target action in the video to be detected according to the target sub-feature sequence; the step of determining the target sub-feature sequence matching the reference feature sequence in the feature sequence to be detected comprises: determining the size of a sliding window according to the length of the reference feature sequence; extracting a sub-feature sequence in the feature sequence to be detected by using the sliding window, and determining the matching degree of the sub-feature sequence and the reference feature sequence; when the matching degree reaches a preset threshold, the sub-feature sequence is determined as the target sub-feature sequence; the step of extracting a sub-feature sequence in the feature sequence to be detected by using the sliding window comprises: determining at least one reference feature frame in the reference feature sequence; determining a feature frame to be detected most similar to the reference feature frame in the feature sequence to be detected, and determining the initial position of the sliding window in the feature sequence to be detected according to the position of the feature frame to be detected; placing the sliding window at the initial position and extracting a sub-feature sequence located in the sliding window.

2. The method of claim 1, wherein, the step of extracting features of image frames in the video to be detected to obtain a feature sequence corresponding to the video to be detected comprises: extracting features of image frames in the video to be detected by using a pre-trained feature extraction model to obtain a feature sequence corresponding to the video to be detected.

3. The method of claim 2, wherein, The method further comprises: obtaining a sample video pair, the sample video pair comprising a first sample video and a second sample video, the first sample video and the second sample video corresponding to the same action; extracting features of image frames in the first sample video and the second sample video by using the feature extraction model to be trained to obtain a first sample feature sequence corresponding to the first sample video and a second sample feature sequence corresponding to the second sample video; for at least one first sample feature in the first sample feature sequence, determining a second sample feature most similar to the first sample feature in the second sample feature sequence to obtain a first matching result, and determining a first sample feature most similar to the second sample feature in the first sample feature sequence to obtain a second matching result; updating the parameters of the feature extraction model according to the difference between the first matching result and the second matching result.

4. The method of claim 1, wherein, The reference video of the target action comprises a plurality of reference videos; the step of determining the size of the sliding window according to the length of the reference feature sequence comprises: comparing a plurality of reference feature sequences corresponding to a plurality of reference videos to determine an outlier feature frame in the reference feature sequence; The outlier feature frame is removed from the reference feature sequence, and a size of the sliding window is determined according to a length of the reference feature sequence after the outlier feature frame is removed.

5. The method of claim 1, wherein, The image frame related to the target action in the video to be detected is determined according to the target sub-feature sequence, including: The start frame and the end frame of the target action in the video to be detected are determined according to the position of the target sub-feature sequence in the feature sequence to be detected.

6. A video detection apparatus characterized by comprising: Including: A video acquisition module is configured to acquire a reference video of a target action and a video to be detected; A feature extraction module is configured to extract features of image frames in the video to be detected to obtain a feature sequence to be detected corresponding to the video to be detected, and extract features of image frames in the reference video to obtain a reference feature sequence corresponding to the reference video; A sequence determination module is configured to determine a target sub-feature sequence matched with the reference feature sequence in the feature sequence to be detected; the target sub-feature sequence is a group sequence composed of features of image frames related to the target action in the feature sequence to be detected; An image determination module is configured to determine image frames related to the target action in the video to be detected according to the target sub-feature sequence; The image determination module is configured to: determine a size of a sliding window according to a length of the reference feature sequence; extract a sub-feature sequence in the feature sequence to be detected by using the sliding window, and determine a matching degree between the sub-feature sequence and the reference feature sequence, when the matching degree reaches a preset threshold, determine the sub-feature sequence as the target sub-feature sequence; The extraction of the sub-feature sequence in the feature sequence to be detected by using the sliding window is configured to: determine at least one reference feature frame in the reference feature sequence; determine a feature frame to be detected most similar to the reference feature frame in the feature sequence to be detected, and determine an initial position of the sliding window in the feature sequence to be detected according to a position of the feature frame to be detected; place the sliding window at the initial position and extract the sub-feature sequence located in the sliding window.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to implement the method of any one of claims 1 to 5.

8. An electronic device, comprising: Including: A processor; A memory is configured to store executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 5 by executing the executable instructions.

Images