Video target detection method, device, electronic device and storage medium

Abstract

The present invention provides a method, device, electronic device and storage medium for detecting video targets. The method comprises: determining a video frame to be detected in a video to be detected, and obtaining an auxiliary video frame in a video sequence that matches the video frame to be detected; determining a frame-level feature vector through a feature extraction network in a video processing model based on the video frame to be detected and the auxiliary video frame; determining an instance-level feature vector through a candidate box generation network in the video processing model based on the frame-level feature vector; performing target detection on the video to be detected according to the instance-level feature vector through a detection head network in the video processing model, and determining the area where the video target is located in different video frames of the video to be detected, thereby achieving robust and accurate detection of targets in the video and reducing distortion in video target detection.

Description

Technical Field

[0001] The present invention relates to image processing technology in videos, and in particular to a method, device, electronic equipment and storage medium for detecting video targets. Background Art

[0002] Deep learning-based classification and recognition has long been a crucial tool for categorizing large amounts of data in various application scenarios. For example, in image and natural language processing applications, large-scale classification and recognition of large amounts of data can quickly and accurately obtain relevant classification prediction results, accelerating the implementation of the application's functions.

[0003] Detecting video targets refers to identifying specific objects (i.e., target objects) in the video being detected and tracking them. Target detection includes both static image and video target detection. Similar to static image target detection, video-based target detection requires accurate identification of the target object's location within the video frame.

[0004] However, the process of object detection in videos in related technologies may introduce noise information, or may cause missed detection due to poor quality of a single video frame, thereby affecting the accuracy of object detection in the video. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a method, device, electronic device and storage medium for detecting video targets, which can perform target detection on the video to be detected based on the feature vector at the instance level, and determine the area where the video target is located in different video frames of the video to be detected, thereby achieving robust and accurate detection of targets in the video.

[0006] The technical solution of the embodiment of the present invention is achieved as follows:

[0007] An embodiment of the present invention provides a method for detecting a video target, comprising:

[0008] Determining a video frame to be detected in the video to be detected, and obtaining an auxiliary video frame in a video sequence that matches the video frame to be detected;

[0009] Based on the video frame to be detected and the auxiliary video frame, determining a frame-level feature vector through a feature extraction network in a video processing model;

[0010] Based on the frame-level feature vector, determining an instance-level feature vector through a candidate box generation network of the video processing model;

[0011] Through the detection head network in the video processing model, target detection is performed on the video to be detected according to the instance-level feature vector, and the area where the video target is located in different video frames of the video to be detected is determined.

[0012] An embodiment of the present invention further provides a device for detecting a video target, comprising:

[0013] An information transmission module is used to determine a video frame to be detected in a video to be detected, and obtain an auxiliary video frame in a video sequence that matches the video frame to be detected;

[0014] An information processing module, configured to determine a frame-level feature vector based on the video frame to be detected and the auxiliary video frame through a feature extraction network in a video processing model;

[0015] The information processing module is configured to determine an instance-level feature vector based on the frame-level feature vector through a candidate box generation network of the video processing model;

[0016] The information processing module is used to perform target detection on the video to be detected based on the instance-level feature vector through the detection head network in the video processing model, and determine the area where the video target is located in different video frames of the video to be detected.

[0017] In the above scheme,

[0018] The information transmission module is used to analyze the video to be detected and obtain the timing information of the video to be detected;

[0019] The information transmission module is used to analyze the video parameters corresponding to the video to be detected according to the timing information of the video to be detected, and obtain the playback duration parameter and the screen area parameter corresponding to the video to be detected;

[0020] The information transmission module is used to extract the video to be detected based on the playback duration parameter and the picture area parameter corresponding to the video to be detected to obtain the video frame to be detected in the video to be detected;

[0021] The information transmission module is configured to randomly extract at least one auxiliary video frame from a video sequence that matches the video frame to be detected in response to the extracted video frame to be detected.

[0022] In the above scheme,

[0023] The information processing module is configured to extract different first frame-level feature vectors from the to-be-detected video frame and the auxiliary video frame respectively through a feature extraction network in a video processing model;

[0024] The information processing module is used to determine the number of channels corresponding to each of the different first frame-level feature vectors;

[0025] The information processing module is configured to determine, based on the number of channels, a corresponding first frame-level feature vector set and a similarity matrix matching the first frame-level feature vector set;

[0026] The information processing module is configured to determine a corresponding second frame-level feature vector set through fusion processing based on the first frame-level feature vector set and the similarity matrix;

[0027] The information processing module is used to determine the frame-level feature vector by linearly transforming the second frame-level feature vector set.

[0028] In the above scheme,

[0029] The information processing module is used to cross-process the video frame to be detected and the auxiliary video frame through the convolution layer and the maximum pooling layer of the feature extraction network in the video processing model, respectively, to obtain downsampling results of the video frame to be detected and the auxiliary video frame;

[0030] The information processing module is configured to normalize the downsampling results of the video frame to be detected and the auxiliary video frame through a fully connected layer of the feature extraction network;

[0031] The information processing module is used to perform deep decomposition processing on the normalized results of downsampling of different image frames through the feature extraction network to determine different first frame level feature vectors.

[0032] In the above scheme,

[0033] The information processing module is configured to process the frame-level feature vectors through a candidate box generation network of the video processing model to determine a set of bounding boxes of candidate object instances;

[0034] The information processing module is configured to determine, based on the bounding box set of the candidate object instance and the frame-level feature vector, a first instance-level feature vector set corresponding to the candidate object instance through a region of interest pooling layer network in the video processing model;

[0035] The information processing module is used to fuse feature vectors of different instance levels in the first instance-level feature vector set to form a second instance-level feature vector set, wherein the fused second instance-level features in the second instance-level feature vector set serve as the instance-level feature vectors input to the detection head network.

[0036] In the above scheme,

[0037] The information processing module is used to determine the corresponding object appearance information and object geometric information of the candidate object instance;

[0038] The information processing module is used to determine the similarity of the object instance level based on the object appearance information and the object geometric information;

[0039] The information processing module is configured to fuse feature vectors of different instance levels in the first instance-level feature vector set according to the similarity of the object instance level to form a second instance-level feature vector set.

[0040] In the above solution, the device further comprises:

[0041] A training module, configured to obtain a first training sample set, wherein the first training sample set is a video processing sample with noise obtained through historical data;

[0042] The training module is configured to perform denoising processing on the first training sample set to form a corresponding second training sample set;

[0043] The training module is configured to process the second training sample set using a video processing model to determine initial parameters of the video processing model;

[0044] The training module is configured to process the second training sample set by the video processing model in response to the initial parameters of the video processing model to determine updated parameters of the video processing model;

[0045] The training module is used to iteratively update the network parameters of the video processing model through the second training sample set according to the update parameters of the video processing model.

[0046] In the above scheme;

[0047] The training module is configured to substitute different training samples in the second training sample set into a loss function corresponding to the video processing model;

[0048] The training module is used to determine the update parameters of the video processing model when the loss function meets the corresponding convergence conditions.

[0049] In the above scheme,

[0050] The information processing module is used to determine the copyrighted video corresponding to the video to be detected;

[0051] The information processing module is configured to determine a set of inter-frame similarity parameters corresponding to the video to be detected and the copyrighted video based on the region where the video target is located in different video frames of the video to be detected;

[0052] The information processing module is used to obtain the number of image frames that reach a similarity threshold in the inter-frame similarity parameter set;

[0053] The information processing module is used to determine the similarity between the video to be detected and the copyrighted video based on the number of image frames that reach a similarity threshold.

[0054] In the above scheme,

[0055] The information processing module is configured to obtain copyright information of the video to be detected when it is determined that the video to be detected is similar to a copyrighted video;

[0056] The information processing module is used to determine the legitimacy of the video to be detected based on the copyright information of the video to be detected and the copyright information of the copyrighted video;

[0057] The information processing module is used to issue a warning message when the copyright information of the video to be detected is inconsistent with the copyright information of the copyrighted video.

[0058] In the above scheme,

[0059] The information processing module is configured to add the video to be detected to the video source when it is determined that the video to be detected is not similar to the copyrighted video;

[0060] The information processing module is used to sort the recall order of all videos to be recommended in the video source;

[0061] The information processing module is used to recommend videos to target users based on the sorting results of the recall order of the videos to be recommended.

[0062] In the above scheme,

[0063] The information processing module is used to determine the detection information corresponding to the video to be detected;

[0064] The information processing module is used to determine the matching degree between the video to be detected and the detection information based on the area where the video target is located in different video frames of the video to be detected;

[0065] The information processing module is used to determine that the video to be detected is compliant when the degree of matching between the video to be detected and the detection information is lower than an alarm threshold, so as to automatically detect the compliance of the area where the video target is located in different video frames of the video to be detected.

[0066] An embodiment of the present invention further provides an electronic device, comprising:

[0067] a memory for storing executable instructions;

[0068] The processor is configured to implement the aforementioned video target detection method when running the executable instructions stored in the memory.

[0069] An embodiment of the present invention further provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the aforementioned video target detection method is implemented.

[0070] The embodiments of the present invention have the following beneficial effects:

[0071] The present invention determines a video frame to be detected in a video to be detected, and obtains an auxiliary video frame in a video sequence that matches the video frame to be detected; based on the video frame to be detected and the auxiliary video frame, a frame-level feature vector is determined through a feature extraction network in a video processing model; based on the frame-level feature vector, an instance-level feature vector is determined through a candidate box generation network of the video processing model; through a detection head network in the video processing model, target detection can be performed on the video to be detected according to the instance-level feature vector, and the area where the video target is located in different video frames of the video to be detected can be determined, thereby achieving robust and accurate detection of targets in the video and reducing distortion in video target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 Schematic diagram of the use environment of the video target detection method provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0074] Figure 3 An optional flowchart of a method for detecting a video target provided by an embodiment of the present invention;

[0075] Figure 4An optional schematic diagram for extracting a video frame to be detected and an auxiliary video frame in an embodiment of the present invention;

[0076] Figure 5 An optional flowchart of a method for detecting a video target provided by an embodiment of the present invention;

[0077] Figure 6 A schematic diagram of an optional process for video similarity determination in an embodiment of the present invention;

[0078] Figure 7 A schematic diagram of a usage scenario of the video target detection method provided by an embodiment of the present invention;

[0079] Figure 8 Schematic diagram of the use process of the video target detection method in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0081] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0082] Before further explaining the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0083] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed can be real-time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0084] 2) Video to be detected: various forms of video information available on the Internet, such as video files and multimedia information presented on the client or smart device.

[0085] 3) Client: The carrier that implements specific functions in the terminal. For example, the mobile client (APP) is the carrier of specific functions in the mobile terminal, such as executing the function of online live broadcast (video streaming) or the function of playing online videos.

[0086] 4) Artificial neural network, also known as neural network (NN), is a mathematical model or computational model that imitates the structure and function of biological neural networks in the field of machine learning and cognitive science, and is used to estimate or approximate functions.

[0087] 5) Model parameters are quantities that use universal variables to establish the relationship between functions and variables. In artificial neural networks, model parameters are usually real number matrices.

[0088] 6) mAP, Mean Average Precision is an evaluation metric for detectors

[0089] 7) Detection head network, the detection head network of target detection mainly includes an object classification branch and a regression branch

[0090] Figure 1 Schematic diagram of the use scenario of the video target detection method provided by the embodiment of the present invention, see Figure 1 , the terminal (including terminal 10-1 and terminal 10-2) is provided with a corresponding client that can perform different functions. Among them, the client is the terminal (including terminal 10-1 and terminal 10-2) that obtains different video information from the corresponding server 200 through the network 300 using different business processes for browsing. The terminal is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, using a wireless link to achieve data transmission. Among them, the types of videos obtained by the terminal (including terminal 10-1 and terminal 10-2) from the corresponding server 200 through the network 300 are different. For example, the terminal (including terminal 10-1 and terminal 10-2) can obtain videos (i.e., videos carrying video information or corresponding video links) from the corresponding server 200 through the network 300, or can obtain corresponding videos that only include text or images from the corresponding server 200 through the network 300 for browsing. The server 200 can store different types of videos. Among them, the compilation environment of different types of videos is no longer distinguished in this application. During this process, the video information pushed to the user's client should be copyright-compliant video information. Therefore, for a large number of videos, it is necessary to determine which videos are similar and further perform compliance detection on the copyright information of similar videos.

[0091] Taking short videos as an example, the video recommendation model provided by the present invention can be applied to short video playback. In short video playback, different short videos from different data sources are usually processed, and finally the recommended video corresponding to the corresponding user is presented on the user interface UI (UserInterface). If the recommended video is a pirated video that does not comply with copyright, it will directly affect the user experience. The background database of video playback receives a large amount of video data from different sources every day, and the different videos obtained for video recommendation to target users can also be called by other applications (for example, the recommendation results of the short video recommendation process are migrated to the long video recommendation process or the news recommendation process). Of course, the video recommendation model that matches the corresponding target user can also be migrated to different video recommendation processes (for example, a web video recommendation process, a mini-program video recommendation process, or a long video client video recommendation process).

[0092] Among them, the video recommendation method provided in the embodiment of the present application is based on artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0093] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0094] In the embodiments of the present application, the artificial intelligence software technologies mainly involved include the above-mentioned speech processing technology and machine learning. For example, it may involve the speech recognition technology (Automatic Speech Recognition, ASR) in speech technology, including speech signal preprocessing, speech signal frequency analysis, speech signal feature extraction, speech signal feature matching / recognition, speech training, etc.

[0095] For example, this may involve machine learning (ML), a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as deep learning, which includes artificial neural networks (ANNs), such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep neural networks (DNNs).

[0096] The structure of the electronic device of the embodiment of the present invention is described in detail below. The electronic device can be implemented in various forms, such as a dedicated terminal with video processing function, such as a gateway, or a server with video processing function, such as the aforementioned Figure 1 Server 200 in. Figure 2 The schematic diagram of the structure of the electronic device provided in the embodiment of the present invention can be understood as follows: Figure 2 Only the exemplary structure of the server is shown, not all structures, and can be implemented as needed. Figure 2 Partial or complete structure shown.

[0097] The electronic device provided in the embodiment of the present invention includes: at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204. The various components in the electronic device 20 are coupled together via a bus system 205. It can be understood that the bus system 205 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 205 is not described in detail. Figure 2 Various buses are labeled as bus system 205.

[0098] The user interface 203 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0099] It is understood that the memory 202 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The memory 202 in the embodiment of the present invention can store data to support the operation of the terminal (such as 10-1). Examples of such data include: any computer program used to operate on the terminal (such as 10-1), such as an operating system and an application program. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program can include various application programs.

[0100] In some embodiments, the video target detection device provided by the embodiments of the present invention can be implemented by combining software and hardware. As an example, the video target detection device provided by the embodiments of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the video target detection method provided by the embodiments of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0101] As an example of a video target detection device provided by an embodiment of the present invention being implemented by a combination of software and hardware, the video target detection device provided by an embodiment of the present invention can be directly embodied as a combination of software modules executed by the processor 201. The software module can be located in a storage medium, and the storage medium is located in the memory 202. The processor 201 reads the executable instructions included in the software module in the memory 202, and combines with the necessary hardware (for example, including the processor 201 and other components connected to the bus 205) to complete the video target detection method provided by the embodiment of the present invention.

[0102] As an example, the processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0103] As an example of a hardware implementation of the video target detection device provided in an embodiment of the present invention, the device provided in an embodiment of the present invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, the video target detection method provided in an embodiment of the present invention can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0104] The memory 202 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic device 20. Examples of such data include any executable instructions for operating on the electronic device 20, such as executable instructions. A program implementing the method for detecting a target from a video according to the embodiment of the present invention may be included in the executable instructions.

[0105] In other embodiments, the video target detection device provided by the embodiment of the present invention can be implemented in a software manner. Figure 2The video target detection device 2020 stored in the memory 202 is shown. The device can be software in the form of a program or plug-in, and includes a series of modules. As an example of a program stored in the memory 202, the device 2020 can include the video target detection device. The video target detection device 2020 includes the following software modules: an information transmission module 2081 and an information processing module 2082. When the software modules in the video target detection device 2020 are read into the RAM and executed by the processor 201, the video target detection method provided by the embodiment of the present invention is implemented. The functions of each software module in the video target detection device 2020 are described below:

[0106] The information transmission module 2081 is used to determine a video frame to be detected in the video to be detected, and obtain an auxiliary video frame in a video sequence that matches the video frame to be detected;

[0107] An information processing module 2082 is configured to determine a frame-level feature vector based on the video frame to be detected and the auxiliary video frame through a feature extraction network in a video processing model;

[0108] The information processing module 2082 is configured to determine an instance-level feature vector based on the frame-level feature vector through a candidate box generation network of the video processing model;

[0109] The information processing module 2082 is used to perform target detection on the video to be detected based on the instance-level feature vector through the detection head network in the video processing model, and determine the area where the video target is located in different video frames of the video to be detected.

[0110] according to Figure 2 In one aspect of the electronic device shown, the present application further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above-mentioned video object detection method.

[0111] Combine Figure 2The electronic device 20 shown illustrates a video object detection method provided by an embodiment of the present invention. Prior to this introduction, the shortcomings of related art are first explained. Related art video object detection based on frame-level fusion can be achieved using FGFA. FGFA first uses an additional optical flow prediction network to generate motion information between multiple frames. This motion information is then used to fuse frame-level features from multiple frames, thereby enhancing the features of the video frame to be detected. Specifically, the feature extraction network extracts feature maps for a single frame. To enhance the features of the current frame, the optical flow prediction network estimates the motion of adjacent frames and the current frame. The feature maps of the adjacent frames are then warped to the current frame based on the optical flow motion. The warped feature maps, along with the current frame's own feature maps, are aggregated through an adaptive weighting network. The resulting aggregated feature maps are then fed into the detection network (in the current frame) to output the detection result for the current frame. However, the quality of the fused features in this process is highly dependent on the accuracy of the additional optical flow network. If the optical flow network's predictions are erroneous, significant noise will be introduced into the feature fusion of the video frame to be detected, reducing recognition accuracy.

[0112] Meanwhile, SELSA (Sensitive Object Detection and Instance-Level Assistance) can be used for video object detection based on object instance fusion. SELSA first generates a series of candidate object instances using a two-stage image object detection algorithm. It then uses a proposed semantic fusion module to fuse the features of each object instance together, thereby enhancing their individual features. However, this approach can lead to missed detections when generating candidate object instances due to poor single-frame video quality, which cannot be corrected through subsequent processing.

[0113] To solve the above defects, see Figure 3 , Figure 3 An optional flow chart of the method for detecting a video target provided by an embodiment of the present invention is provided. It can be understood that: Figure 3 The steps shown can be performed by various servers running the video target detection device, such as a dedicated terminal, server or server cluster with video processing function. The video target detection method provided by the embodiment of the present invention can be used for non-real-time video target detection, such as content analysis (including various video types such as TV series, movies, short videos, etc.), target object detection and recognition, etc. Figure 3 The steps shown are explained.

[0114] Step 301: The video target detection apparatus determines a video frame to be detected in a video to be detected, and obtains an auxiliary video frame in a video sequence that matches the video frame to be detected.

[0115] In some embodiments of the present invention, determining a video frame to be detected in a video to be detected and obtaining an auxiliary video frame in a video sequence that matches the video frame to be detected can be achieved by:

[0116] The video to be detected is parsed to obtain the timing information of the video to be detected; based on the timing information of the video to be detected, the video parameters corresponding to the video to be detected are parsed to obtain the playback duration parameters and screen area parameters corresponding to the video to be detected; based on the playback duration parameters and screen area parameters corresponding to the video to be detected, the video to be detected is extracted to obtain the video frame to be detected in the video to be detected; in response to the extracted video frame to be detected, at least one auxiliary video frame is randomly extracted from the video sequence that matches the video frame to be detected. Figure 4 The present invention provides an optional schematic diagram of extracting the video frame to be detected and the auxiliary video frame in an embodiment of the present invention. As the video plays, the displayed image area changes over time. Different video targets are present in the displayed image area. By detecting the video targets, the area where the video targets are located in different video frames of the video to be detected can be determined. The detection results of the video targets can then be used to determine whether the video to be detected is compliant or meets copyright information requirements, thereby preventing user-uploaded videos from being broadcast illegally.

[0117] Step 302: The video target detection device determines a frame-level feature vector through a feature extraction network in a video processing model based on the video frame to be detected and the auxiliary video frame.

[0118] In some embodiments of the present invention, determining a frame-level feature vector by a feature extraction network in a video processing model based on the video frame to be detected and the auxiliary video frame can be achieved by:

[0119] Through the feature extraction network in the video processing model, different first frame-level feature vectors are extracted from the video frame to be detected and the auxiliary video frame respectively; the number of channels corresponding to each of the different first frame-level feature vectors is determined; based on the number of channels, the corresponding first frame-level feature vector set and the similarity matrix matching the first frame-level feature vector set are determined; according to the first frame-level feature vector set and the similarity matrix, the corresponding second frame-level feature vector set is determined through fusion processing; the frame-level feature vector is determined through linear transformation processing of the second frame-level feature vector set. Given a video frame to be detected, the present invention first randomly samples n-1 auxiliary video frames in the video where the detection frame is located. These auxiliary video frames can provide rich spatiotemporal information for the video frame to be detected. Then, a feature extraction network is used to extract a series of frame-level features F{F1, F2.....F n}, where f i ∈F represents the frame-level features extracted on the i-th frame. Since each feature in F has d channels, each feature in F can be separated by channel to obtain n*d channel-level features

[0120] Furthermore, in the frame-level fusion stage, the present invention calculates F c A similarity matrix s F To express F c The apparent similarity between each feature in . Then, for F c The i-th feature F in i c , the present invention is based on s F Come and F c All features in F are fused into i c To generate the corresponding enhanced feature F i e Here, the generated enhanced features can be expressed as Especially, F e The i-th enhanced feature F in e i It is calculated by formula 1:

[0121]

[0122] where θ(·) represents a linear transformation function implemented by a fully connected layer. Indicates F i c and F jc The apparent similarity between them is calculated as shown in Formula 2:

[0123]

[0124] where a i,j It's F i c and F j c The dot product result between is shown in Formula 3:

[0125]

[0126] φ(·) and are two linear transformation functions that have the same function as θ(·). After the frame-level feature fusion, the i-th feature F i e ∈F e The information present in is propagated to F e Among the other features in F, e The features in can obtain rich information existing in the frame-level features from other frames.

[0127] Step 303: The video target detection device determines an instance-level feature vector based on the frame-level feature vector through the candidate box generation network of the video processing model.

[0128] In some embodiments of the present invention, determining the instance-level feature vector based on the frame-level feature vector by the candidate box generation network of the video processing model can be achieved by:

[0129] The frame-level feature vectors are processed by the candidate box generation network of the video processing model to determine a set of bounding boxes of candidate object instances; based on the set of bounding boxes of candidate object instances and the frame-level feature vectors, a first instance-level feature vector set corresponding to the candidate object instances is determined by the region of interest pooling layer network (ROI pooling layer network) in the video processing model; different instance-level feature vectors in the first instance-level feature vector set are fused to form a second instance-level feature vector set, wherein the fused second instance-level features in the second instance-level feature vector set are used as instance-level feature vectors input to the detection head network. e Input a candidate box extraction network (RPN) to obtain a series of bounding boxes of candidate object instances B = {B1, B2....B mHere m represents the number of generated object instances, and each bounding box in B contains the location of the center point of the object instance and the scale of the object instance. Then these bounding boxes B and the enhanced frame-level features F e It is input into a RoI layer to extract a series of object instance level features Q = {Q1, Q2....Q m}. Specifically, the ROI pooling layer can achieve significant acceleration of training and testing, and improve detection accuracy. The region of interest pooling layer network (ROIpooling layer network) has two inputs: 1) fixed-size feature maps obtained from a deep network with multiple convolution kernel pooling; 2) an N*5 matrix representing all ROIs, where N represents the number of ROIs. The first column represents the image index, and the remaining four columns represent the remaining upper left and lower right corner coordinates. During the processing, the region of interest pooling layer network (ROI pooling layer network) first maps the ROI to the corresponding position of the feature map according to the first instance-level feature vector of the input; then divides the mapped area into sections of the same size, where the number of sections is the same as the output dimension; finally, a max pooling operation is performed on each section; in this process, since the size of the output feature maps does not depend on the size of the ROI and convolution feature maps, the region of interest pooling layer network (ROI pooling layer network) can effectively improve the processing speed, reduce the time users wait for video review, and improve the user experience.

[0130] Furthermore, in one embodiment of the present application, in order to avoid target distortion during video target detection, the corresponding object appearance information and object geometry information of the candidate object instance can be determined during the fusion process; based on the object appearance information and object geometry information, the similarity of the object instance level can be determined; according to the similarity of the object instance level, the feature vectors of different instance levels in the first instance level feature vector set are fused to form a second instance level feature vector set. Specifically, the features in Q are fused with each other to obtain the final enhanced object instance level features The specific formula is as follows:

[0131]

[0132] where γ(·) represents a linear transformation function, It means Q k and Q i The instance-level similarity between objects.

[0133] For characterizing an object instance, in addition to its appearance information, its geometric information is also very important. Therefore, in the object instance level fusion module, the present invention proposes to use both the appearance information Q and the geometric information B of the object to measure the similarity between object instances. The similarity refers to Formula 5:

[0134]

[0135] where Z k,l Represents Q k and Q l The apparent similarity between k,l is the kth bounding box B k and the lth bounding box B l The geometric similarity between Z k,l Calculated by the following formula 6:

[0136] z k,l =<ξ(Q k ),ζ(Q l )> Formula 6

[0137] Where ξ(·) and ζ(·) are two linear transformation functions implemented by the fully connected layer. Since different objects may appear in the same position in different frames, the scale information of the object (that is, the width w and height h of the object) is more reliable than the position information of the object when measuring geometric similarity. Therefore, the present invention proposes to use the scale information of the object to calculate the bounding box B k and bounding box B l The similarity between k,l Refer to Formula 7:

[0138]

[0139] Where ψ(·) is the same linear transformation function as ξ(·) and ζ(·). σ(·) is an embedding function that uses trigonometric functions of different frequencies to transform the similarity value r in the low-dimensional space into k,l Mapped to a high-dimensional space, Formula 8 is as follows:

[0140]

[0141] where d geo Refers to r k,l Each value in is mapped to the dimension of the high-dimensional space, so the final r k,l The dimension is d geo *3. By using both geometric and appearance similarities in the fusion process, the proposed DSFNet can alleviate the problem of information distortion caused by noise in the fusion process. Finally, the enhanced feature Qe The features belonging to the current frame are input to the detection head network for final object classification and bounding box regression.

[0142] Step 304: The video target detection device performs target detection on the video to be detected based on the instance-level feature vector through the detection head network in the video processing model, and determines the area where the video target is located in different video frames of the video to be detected.

[0143] Continue to combine Figure 2 The electronic device 20 shown illustrates a method for detecting a video target provided by an embodiment of the present invention, see Figure 5 , Figure 5 An optional flow chart of the method for detecting a video target provided by an embodiment of the present invention is provided. It can be understood that: Figure 5 The steps shown can be performed by various servers running video processing functions, such as dedicated terminals, servers, or server clusters with video processing functions. The trained video processing can be deployed in the server to detect the similarity of uploaded videos to determine whether the copyright information of the video is compliant. Of course, the video processing model needs to be trained before deployment, which specifically includes the following steps:

[0144] Step 501: Acquire a first training sample set, wherein the first training sample set is a video processing sample with noise obtained through historical data.

[0145] Step 502: Perform denoising processing on the first training sample set to form a corresponding second training sample set.

[0146] Step 503: Process the second training sample set using a video processing model to determine initial parameters of the video processing model.

[0147] Step 504: In response to the initial parameters of the video processing model, the second training sample set is processed by the video processing model to determine the updated parameters of the video processing model.

[0148] Specifically, different training samples in the second training sample set may be substituted into the loss function corresponding to the video processing model; and the update parameters of the video processing model corresponding to when the loss function satisfies the corresponding convergence condition are determined.

[0149] Step 505: According to the updated parameters of the video processing model, the network parameters of the video processing model are iteratively updated through the second training sample set.

[0150] Among them, when the video processing model is trained, the video processing model approaches the correct trend through loss functions such as cross entropy until the loss function reaches the corresponding convergence condition.

[0151] In some embodiments of the present invention, the feature extraction network in the video processing model can also use ResNet-101 or ResNeXt-101. Among them, the ResNeXt-101 model uses user-labeled pictures on Instagram as a pre-training data set, which can save the huge cost of manually labeling data. Moreover, only fine-tuning is required during use, and the performance of the model exceeds the SOTA level of the ImageNet task, which is conducive to the large-scale use of the video processing model of this application in users' mobile terminals.

[0152] Furthermore, the parameters of the video processing model are initialized by weights pre-trained on the ImageNet classification dataset, and the datasets used are video datasets and picture datasets. Each input video frame image is first scaled to an image frame with a shorter side less than 600 pixels. The loss function used by the DSFNet proposed in the present invention is cross entropy loss, and SGD is used as the optimizer for training on 8 graphics cards, where the number of samples used for one gradient descent is 8. The DSFNet proposed in the present invention was trained for a total of 247k iterations, and the initial learning rate was set to 2.5*10 -4 , the learning rate decreases by 10% at the 109kth and 219th iterations, respectively. During the testing phase, for each video frame to be detected, the present invention randomly samples n-1=20 frames as auxiliary video frames for video object detection. In addition, during training and testing, the present invention uses RPN to extract 300 candidate object instances in each frame.

[0153] Step 506: Deploy the trained video processing model.

[0154] Therefore, the corresponding video target detection method can be executed through the deployed trained video processing model (which can be deployed in the server or cloud server of the video client operator) to realize the detection of the video uploaded by the user.

[0155] Among them, continue to refer to Figure 6 , Figure 6 This is a schematic diagram of an optional process for determining video similarity in an embodiment of the present invention, specifically including:

[0156] Step 601: Determine the copyrighted video corresponding to the video to be detected.

[0157] Step 602: Based on the regions where the video target is located in different video frames of the video to be detected, determine a set of inter-frame similarity parameters corresponding to the video to be detected and the copyrighted video.

[0158] Step 603: Obtain the number of image frames that reach a similarity threshold in the inter-frame similarity parameter set.

[0159] Step 604: Determine the similarity between the video to be detected and the copyrighted video based on the number of image frames that reach the similarity threshold.

[0160] Step 605: Based on the similarity between the video to be detected and the copyrighted video and the set similarity threshold, determine whether they are similar. If so, execute step 606; otherwise, execute step 607.

[0161] Step 606: Determine the video similarity through similarity determination.

[0162] When it is determined that the video to be detected is similar to a copyrighted video, the copyright information of the video to be detected is obtained; the legality of the video to be detected is determined based on the copyright information of the video to be detected and the copyright information of the copyrighted video; if the copyright information of the video to be detected is inconsistent with the copyrighted video, a warning message is issued. In this way, it is possible to determine whether the copyrighted video has been pirated by the location of the video target in different video frames of the video to be detected.

[0163] Step 607: Determine that the videos are different through similarity determination.

[0164] If the video to be detected is determined to be dissimilar to a copyrighted video, the video to be detected is added to the video source; the recall order of all videos to be recommended in the video source is sorted; and based on the sorted recall order of the videos to be recommended, videos are recommended to the target user. This implementation identifies the corresponding copyrighted videos based on the location of the video target in different video frames of the video to be detected and recommends them to the user, enriching the user's video viewing options.

[0165] In some embodiments of the present invention, detection information corresponding to the video to be detected can also be determined; based on the area where the video target is located in different video frames of the video to be detected, the degree of matching between the video to be detected and the detection information is determined; when the degree of matching between the video to be detected and the detection information is lower than the alarm threshold, the compliance of the video to be detected is determined to realize automatic detection of the compliance of the area where the video target is located in different video frames of the video to be detected, thereby reducing manual participation in the video review process, improving the rate of video compliance detection, reducing the cost of detection, and reducing the user's waiting time.

[0166] As the number of videos on video servers continues to increase, the copyright information of the videos can be stored in the blockchain network or cloud server to realize the judgment of video similarity. Among them, the embodiments of the present invention can be implemented in combination with cloud technology or blockchain network technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or a local area network to realize the calculation, storage, processing and sharing of data. It can also be understood as a general term for network technology, information technology, integration technology, management platform technology and application technology based on cloud computing business model applications. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. Therefore, cloud technology needs to be supported by cloud computing.

[0167] It should be noted that cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, the resources in the "cloud" appear to be infinitely scalable and can be accessed at any time, used on demand, and expanded at any time, with a pay-per-use fee. As a provider of cloud computing's basic capabilities, a cloud computing resource pool platform, often referred to as Infrastructure as a Service (IaaS), is established. Various types of virtual resources are deployed within the resource pool for external customers to choose from. The cloud computing resource pool primarily includes computing devices (which can be virtualized machines, including operating systems), storage devices, and network devices.

[0168] The following describes the detection method for video targets provided by this application, taking the implementation environment of copyright detection of short videos as an example. Figure 7 Schematic diagram of the use scenario of the video target detection method provided by the embodiment of the present invention, see Figure 7 , the terminal (including terminal 10-1 and terminal 10-2) is provided with a client of software capable of displaying the corresponding short video, such as a client or plug-in for short video playback. The user can obtain the first video and display it through the corresponding client; the terminal is connected to the short video server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission. Of course, users can also upload videos through the terminal for other users in the network to watch. During this process, the operator's video server needs to detect the videos uploaded by the user, compare and analyze different video information, determine whether the copyright of the videos uploaded by the user is compliant, and recommend compliant videos to different users to prevent the user's short videos from being pirated.

[0169] Among them, reference Figure 8 , Figure 8 The following is a schematic diagram of the process of using the video object detection method according to an embodiment of the present invention, which specifically includes the following steps:

[0170] Step 801: Determine a video frame of a short video to be detected, and obtain an auxiliary video frame in a video sequence that matches the video frame to be detected.

[0171] Step 802: Extract frame-level feature vectors from the video frame to be detected and the auxiliary video frame through the feature extraction network in the video processing model.

[0172] Step 803: Perform enhancement processing on the frame-level feature vector.

[0173] Step 804: The enhanced frame-level feature vector is processed by the candidate box generation network of the video processing model to form an instance-level feature vector.

[0174] Step 805: Enhance the instance-level feature vector to achieve classification and regression processing of the enhanced instance-level feature vector through the detection head network to achieve video target detection.

[0175] Step 806: Based on the regions where the video target is located in different video frames of the short video to be detected, determine a set of inter-frame similarity parameters corresponding to the video to be detected and the copyrighted video.

[0176] Step 807: Determine the similarity between the video to be detected and the copyrighted video, and recommend short videos that are dissimilar to the copyrighted video to different users of the client.

[0177] In some embodiments of the present invention, the video target detection method provided by the present application can robustly and accurately detect targets in the video. The accuracy on the public dataset ImageNet VID verification set is shown in Table 1. Compared with the benchmark reference method Faster R-CNN, the present invention has a significant improvement in the accuracy of target recognition in mAP (mean Average Precision, that is, the average value of AP of each category) (+9.4%). Compared with other current state-of-the-art methods, the video target detection method provided by the present application can achieve the highest accuracy whether it is based on ResNet-101 or ResNeXt-101 as the backbone.

[0178] Table 1

[0179]

[0180] Beneficial technical effects: The present invention determines a video frame to be detected in a video to be detected, and obtains an auxiliary video frame in a video sequence that matches the video frame to be detected; based on the video frame to be detected and the auxiliary video frame, a frame-level feature vector is determined through a feature extraction network in a video processing model; based on the frame-level feature vector, an instance-level feature vector is determined through a candidate box generation network of the video processing model; through a detection head network in the video processing model, target detection can be performed on the video to be detected according to the instance-level feature vector, and the area where the video target is located in different video frames of the video to be detected can be determined, thereby achieving robust and accurate detection of targets in the video and reducing distortion in video target detection.

[0181] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting a video target, characterized in that: The method comprises: Determining a video frame to be detected in the video to be detected, and obtaining an auxiliary video frame in a video sequence that matches the video frame to be detected; Based on the video frame to be detected and the auxiliary video frame, determining a frame-level feature vector through a feature extraction network in a video processing model; Based on the frame-level feature vectors, determining a first instance-level feature vector set corresponding to the candidate object instance through a candidate box generation network of the video processing model; Determining object instance-level similarity based on corresponding object appearance information and object geometry information of the candidate object instances; According to the similarity of the object instance level, fusing feature vectors of different instance levels in the first instance level feature vector set to form a second instance level feature vector set; using the fused second instance-level features in the second instance-level feature vector set as instance-level feature vectors; Through the detection head network in the video processing model, target detection is performed on the video to be detected according to the instance-level feature vector, and the area where the video target is located in different video frames of the video to be detected is determined.

2. The method according to claim 1, characterized in that The step of determining a video frame to be detected in a video to be detected and obtaining an auxiliary video frame in a video sequence that matches the video frame to be detected includes: Analyzing the video to be detected to obtain timing information of the video to be detected; Parsing the video parameters corresponding to the video to be detected according to the timing information of the video to be detected, and obtaining the playback duration parameter and the screen area parameter corresponding to the video to be detected; Extracting the video to be detected based on the playback duration parameter and the screen area parameter corresponding to the video to be detected to obtain a video frame to be detected in the video to be detected; In response to the extracted video frame to be detected, at least one auxiliary video frame is randomly extracted from a video sequence that matches the video frame to be detected.

3. The method according to claim 1, characterized in that The determining of a frame-level feature vector based on the to-be-detected video frame and the auxiliary video frame by a feature extraction network in a video processing model includes: Extracting different first frame-level feature vectors from the to-be-detected video frame and the auxiliary video frame respectively through a feature extraction network in a video processing model; Determining the number of channels corresponding to each of the different first frame-level feature vectors; Determining, based on the number of channels, a corresponding first frame-level feature vector set and a similarity matrix matching the first frame-level feature vector set; Determining a corresponding second frame-level feature vector set through fusion processing according to the first frame-level feature vector set and the similarity matrix; The frame-level feature vector is determined by performing a linear transformation on the second frame-level feature vector set.

4. The method according to claim 3, characterized in that The extracting different first frame level feature vectors from the to-be-detected video frame and the auxiliary video frame respectively through a feature extraction network in the video processing model includes: Cross-processing the video frame to be detected and the auxiliary video frame respectively through the convolution layer and the maximum pooling layer of the feature extraction network in the video processing model to obtain downsampling results of the video frame to be detected and the auxiliary video frame; Normalizing the downsampling results of the video frame to be detected and the auxiliary video frame through the fully connected layer of the feature extraction network; The normalized results of downsampling of different image frames are subjected to deep decomposition processing by the feature extraction network to determine different first frame level feature vectors.

5. The method according to claim 1, wherein The determining, based on the frame-level feature vectors, a first instance-level feature vector set corresponding to the candidate object instance through a candidate box generation network of the video processing model includes: Processing the frame-level feature vectors through a candidate box generation network of the video processing model to determine a set of bounding boxes of the candidate object instances; Based on the bounding box set of the candidate object instance and the frame-level feature vector, a first instance-level feature vector set corresponding to the candidate object instance is determined through a region of interest pooling layer network in the video processing model.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire a first training sample set, wherein the first training sample set is a video processing sample with noise obtained through historical data; performing denoising processing on the first training sample set to form a corresponding second training sample set; Processing the second training sample set using a video processing model to determine initial parameters of the video processing model; In response to initial parameters of the video processing model, processing the second training sample set by the video processing model to determine updated parameters of the video processing model; According to the updated parameters of the video processing model, the network parameters of the video processing model are iteratively updated through the second training sample set.

7. The method according to claim 6, characterized in that The step of processing the second training sample set by the video processing model in response to the initial parameters of the video processing model to determine updated parameters of the video processing model; Substituting different training samples in the second training sample set into the loss function corresponding to the video processing model; Determine the update parameters of the video processing model corresponding to when the loss function meets the corresponding convergence condition.

8. The method according to claim 1, characterized in that The method further comprises: Determining a copyrighted video corresponding to the video to be detected; Determining a set of inter-frame similarity parameters corresponding to the video to be detected and the copyrighted video based on regions where the video target is located in different video frames of the video to be detected; Obtaining the number of image frames that reach a similarity threshold in the inter-frame similarity parameter set; Based on the number of image frames that reach a similarity threshold, the similarity between the video to be detected and the copyrighted video is determined.

9. The method according to claim 8, characterized in that The method further comprises: When it is determined that the video to be detected is similar to the copyrighted video, obtaining copyright information of the video to be detected; Determining the legitimacy of the video to be detected based on the copyright information of the video to be detected and the copyright information of the copyrighted video; When the copyright information of the video to be detected is inconsistent with the copyright information of the copyrighted video, a warning message is issued.

10. The method according to claim 8, characterized in that The method further comprises: When it is determined that the video to be detected is not similar to the copyrighted video, adding the video to be detected to the video source; Sorting the recall order of all videos to be recommended in the video source; Recommend videos to target users based on the sorting results of the recall order of the videos to be recommended.

11. The method according to claim 1, characterized in that The method further comprises: Determining detection information corresponding to the video to be detected; Determining a degree of matching between the video to be detected and the detection information based on regions where the video target is located in different video frames of the video to be detected; When the matching degree between the video to be detected and the detection information is lower than an alarm threshold, the video to be detected is determined to be compliant, so as to automatically detect the compliance of the area where the video target is located in different video frames of the video to be detected.

12. A video target detection device, characterized in that: The device comprises: An information transmission module is used to determine a video frame to be detected in a video to be detected, and obtain an auxiliary video frame in a video sequence that matches the video frame to be detected; An information processing module, configured to determine a frame-level feature vector based on the video frame to be detected and the auxiliary video frame through a feature extraction network in a video processing model; The information processing module is configured to determine, based on the frame-level feature vectors, a first instance-level feature vector set corresponding to the candidate object instance through a candidate box generation network of the video processing model; The information processing module is configured to determine the similarity at the object instance level based on the corresponding object appearance information and object geometry information of the candidate object instances; The information processing module is configured to fuse feature vectors of different instance levels in the first instance-level feature vector set according to the similarity of the object instance level to form a second instance-level feature vector set; The information processing module is configured to use the fused second instance-level features in the second instance-level feature vector set as instance-level feature vectors; The information processing module is used to perform target detection on the video to be detected based on the instance-level feature vector through the detection head network in the video processing model, and determine the area where the video target is located in different video frames of the video to be detected.

13. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; A processor, configured to implement the video target detection method according to any one of claims 1 to 11 when running the executable instructions stored in the memory.

14. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the video target detection method according to any one of claims 1 to 11 is implemented.

Images