Video behavior detection, response method and device, equipment, and medium

Abstract

The present application discloses a video behavior detection and response method and apparatus, equipment, and medium, wherein the detection method includes: extracting a space-time feature map from a video stream; pooling the space-time feature map into a one-dimensional feature map that only retains the timing characteristics of the video stream, evaluating the first probability score that the control area of ​​each timing point in the one-dimensional feature map belongs to the target timing behavior, and concurrently evaluating the offset of the candidate box in the timing point of the one-dimensional feature map and the second probability score that the candidate box contains the target timing behavior; optimizing several candidate boxes according to the first probability score and the second probability score; pooling the candidate boxes into multiple feature maps and outputting them to a fully connected layer, which completes video behavior classification and regression, and outputs the classification result data of the video stream. The present application can improve the accuracy of artificial neural networks in identifying video behaviors and significantly improve the online detection effect of video behaviors.

Description

Technical Field

[0001] The present application relates to image recognition technology, and specifically to a video behavior detection and response method and its respective corresponding neural network implementation device, electronic device, and non-volatile storage medium. Background Art

[0002] The technology of using neural networks for video behavior recognition is becoming increasingly mature. The most popular video behavior recognition method is implemented using the R-C3D algorithm. R-C3D is the abbreviation of Region Convolutional 3D Network for TemporalActivity Detection. This algorithm is mainly based on the C3D network and draws on the ideas of Faster RCNN. For any input video L, it first performs proposal (proposal, aiming to provide temporal candidate boxes), then performs pooling (3D-pooling), and finally performs classification and regression operations.

[0003] R-C3D mainly includes feature extraction network, temporal candidate box proposal module and behavior recognition network. For the input video, the C3D multi-layer convolutional network is first used to extract features that are subsequently shared by the temporal candidate box proposal module and the behavior recognition network. Then, the temporal candidate box proposal module is used to select several candidate boxes containing target behaviors. Finally, the behavior recognition network classifies the target behaviors in these candidate boxes to achieve behavior recognition.

[0004] The implementation of the R-C3D algorithm can perform end-to-end detection on videos of any length and behaviors of any length. Its detection speed is very fast, up to 5 times that of other similar networks. It has been tested on a variety of different data sets and the results are good. It is universal and widely welcomed in the industry.

[0005] However, the R-C3D algorithm is implemented based on the anchor regression strategy, which generates candidate boxes by assuming multiple multi-anchor areas of equal length in time. In this way, the background time segments are often not effectively suppressed, which is not conducive to the detection of variable-duration behaviors. Therefore, RC3D-like methods are usually slightly inferior to non-end-to-end methods on the same dataset. Summary of the invention

[0006] The purpose of this application is to provide an efficient video behavior detection method, and on this basis, to provide a video behavior response method applying the method, and at the same time to provide a neural network implementation device, as well as an electronic device and a non-volatile storage medium corresponding to the video response method.

[0007] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0008] A video behavior detection method provided for one of the purposes of this application comprises the following steps:

[0009] Extract features from the video stream to obtain a spatiotemporal feature map containing spatial and temporal feature information;

[0010] Pooling the spatiotemporal feature map into a one-dimensional feature map that only retains the temporal features of the video stream, evaluating a first probability score that the control area of ​​each temporal point in the one-dimensional feature map belongs to the target temporal behavior, and in parallel, evaluating the offset of a series of candidate boxes in the temporal points of the one-dimensional feature map and a second probability score that the candidate box contains the target temporal behavior;

[0011] Selecting a plurality of candidate boxes with higher confidence levels according to the first probability score and the second probability score;

[0012] Using the spatiotemporal feature map as a shared feature, the candidate frames are pooled into multiple feature maps with the same dimension and output to the fully connected layer, which completes the video behavior classification and regression and outputs the classification result data of the video stream.

[0013] In some embodiments, the step of pooling the spatiotemporal feature map into a one-dimensional feature map that only retains the temporal features of the video stream includes: downsampling the spatial resolution of the spatiotemporal feature map to a size of 1x1 by 3D pooling to obtain the one-dimensional feature map.

[0014] In some embodiments, the step of evaluating a first probability score that the control region of each time sequence point in the one-dimensional feature graph belongs to the target time sequence behavior includes:

[0015] Slide a time sequence window of a preset length on the one-dimensional feature map to obtain the features of each time sequence point and its context information;

[0016] The features corresponding to the time sequence window are fully connected to the preset foreground and background binary classification network to determine the first probability score that the corresponding time sequence point control area belongs to the target time sequence behavior.

[0017] In some embodiments, the step of evaluating the offset of a series of candidate boxes in the time points of the one-dimensional feature graph and the second probability score that the candidate boxes contain the target time sequence behavior includes:

[0018] Slide a time sequence window of a preset length on the one-dimensional feature map to obtain the features of each time sequence point and its context information;

[0019] The features corresponding to the timing window are fully connected to a preset two-classification network that is independent of the behavior category, and the offset of the candidate box corresponding to the corresponding timing point control area and the second probability score that the candidate box contains the target timing behavior are determined.

[0020] In some embodiments, the step of selecting several candidate boxes with higher confidence levels according to the probability scores includes:

[0021] Counting the first probability scores corresponding to the candidate boxes to obtain the first probability total scores of the candidate boxes;

[0022] Taking a weighted average of the first probability total score and the second probability score corresponding to each candidate frame system to obtain a weighted total score;

[0023] The weighted total scores are sorted, and several candidate boxes with higher confidence are selected.

[0024] In some embodiments, the step of pooling the candidate boxes into multiple feature maps with the same dimension and outputting them to the fully connected layer includes:

[0025] According to each selected candidate frame, extracting a corresponding feature map to be pooled from the space-time feature map;

[0026] The feature map to be pooled is divided into multiple spatial-temporal sub-feature maps, and the temporal features corresponding to the candidate box are retained between the multiple spatial-temporal sub-features;

[0027] Applying a maximum pooling function to pool the space-time sub-feature map to obtain a result feature map;

[0028] The resulting feature map obtained by pooling is output to the fully connected layer.

[0029] In some embodiments, the step of pooling the candidate boxes into multiple feature maps with the same dimension and outputting them to the fully connected layer includes:

[0030] Extracting the feature maps to be pooled belonging to the continuous segments before and after the selected candidate frames from the space-time feature map according to the time sequence length occupied by each candidate frame;

[0031] The feature map to be pooled of each fragment is divided into space-time sub-feature maps, and the temporal features corresponding to the candidate box are retained between multiple space-time sub-features;

[0032] Applying a maximum pooling function to pool the spatial-temporal sub-feature maps of each segment respectively and unifying them into a result feature map;

[0033] The resulting feature maps obtained by pooling are connected and output to the fully connected layer.

[0034] In a further embodiment, the step of pooling the candidate boxes into multiple feature maps with the same dimension and outputting them to the fully connected layer includes: based on the spatial-temporal sub-feature maps and the result feature maps of the respective fragments, while retaining their structured coding features, performing correlation calculations on the features at the same spatial point along the temporal dimension to obtain corresponding coding features for output to the fully connected layer.

[0035] A video behavior response method provided for one of the purposes of the present application comprises the following steps:

[0036] Get the video stream of the live broadcast user;

[0037] Importing the video stream into an artificial neural network to implement video behavior recognition, the artificial neural network executes the video behavior detection method and generates classification result data of the video stream accordingly;

[0038] Based on the classification result data output by the artificial neural network, a preset and corresponding notification message is output to the live broadcast room users.

[0039] A neural network implementation device provided to meet one of the purposes of the present application includes:

[0040] A feature extraction network is used to extract features from the video stream to obtain a spatiotemporal feature map containing spatial and temporal feature information;

[0041] A temporal candidate network, which pools the spatiotemporal feature map into a one-dimensional feature map that only retains the temporal features of the video stream, and includes a temporal position perception module and a temporal candidate box suggestion module, wherein the temporal position perception module is used to evaluate the first probability score that the control area of ​​each temporal point in the one-dimensional feature map belongs to the target temporal behavior; the temporal candidate box suggestion module, which runs in parallel with the temporal position perception module, is used to evaluate the offset of a series of candidate boxes in the temporal points of the one-dimensional feature map and the second probability score that the candidate box contains the target temporal behavior;

[0042] A temporal candidate box decision network, used for selecting a plurality of candidate boxes with higher confidence according to the first probability score and the second probability score;

[0043] The behavior recognition network is used to use the spatiotemporal feature map as a shared feature, pool the candidate frames into multiple feature maps with the same dimension and output them to the fully connected layer, which completes the video behavior classification and regression and outputs the classification result data of the video stream.

[0044] An electronic device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the video behavior response method.

[0045] A non-volatile storage medium is provided to meet one of the purposes of the present application, which stores a computer program implemented according to the video behavior response method in the form of computer-readable instructions. When the computer program is called by a computer, the steps included in the method are executed.

[0046] Compared with the prior art, this application has the following advantages:

[0047] First of all, the present application makes improvements based on the R-C3D algorithm. In the process of processing the video stream, the space-time feature map is first used to sample a one-dimensional feature map. Then, on the one hand, the temporal features of the one-dimensional feature map are used to determine the temporal candidate frame from the space-time feature map and evaluate the probability score of the target behavior. On the other hand, the probability scores of the temporal positions where the target behavior may exist are evaluated synchronously and in parallel based on the one-dimensional feature map of the video stream. Finally, the candidate frame is recommended using the probability score, thereby realizing the technical means of multi-task learning. It is not necessary to rely on the anchor mechanism of R-C3D. Instead, the temporal candidate frame is generated based on temporal position perception. Therefore, the background temporal fragments in the video stream can be more effectively suppressed, the recall rate of the temporal candidate frame generation and the accuracy of boundary positioning can be improved, and video behavior recognition can be more efficient and accurate.

[0048] Secondly, based on the aforementioned advantages, when the present application is applied to the field of live webcasting for real-time video stream behavior recognition, user behavior in the video stream can be identified more quickly and accurately, and further corresponding responses can be made quickly based on user behavior.

[0049] In addition, this application enriches the technical content of R-C3D based on the improvements made to R-C3D, and essentially proposes a new artificial neural network architecture, which has universal technical architecture and is widely applicable to occasions requiring video behavior recognition and response. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 A schematic diagram of a network architecture suitable for deploying the present application;

[0052] Figure 2 A schematic diagram of the structure of an artificial neural network constructed according to the video behavior detection method of the present application;

[0053] Figure 3 A flowchart of a typical embodiment of the video behavior detection method of the present application;

[0054] Figure 4 and Figure 5 Both Figure 3 The flowchart of the specific steps included in step S12 is shown in FIG. Figure 4 and Figure 5 Belong to the process of parallel implementation;

[0055] Figure 6 for Figure 3 A flow chart of the specific steps included in step S13;

[0056] Figure 7 This is a schematic diagram of the principles corresponding to the application of multiple encoding methods in step S14 of the present application, wherein Proposal refers to a temporal candidate box (candidate box), and 3D Feature Map refers to a spatial-temporal feature map;

[0057] Figure 8 , Fig. 9 , Fig.10 They are Figure 3 Schematic diagram of the process embodied by various specific implementations of step S14;

[0058] Fig.11 When the video behavior detection method of this application is applied to the experiment, the statistics of the candidate boxes in normal time series are obtained by comparing with the R-C3D method to prove the effectiveness;

[0059] Fig.12 The recall curve obtained when the video behavior detection method of this application is applied in the experiment and compared with the R-C3D method to demonstrate the effectiveness;

[0060] Fig.13 Schematic diagram of a typical embodiment of the video behavior response method of the present application. DETAILED DESCRIPTION

[0061] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0062] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0063] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0064] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0065] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0066] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0067] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud, and is mainly used to deploy the artificial neural network of this application. It is only a server in a logical concept, representing the entire service cluster used to implement the artificial neural network. Through this logical server, services are provided to related third-party servers and terminal devices such as the smart phone 81 and personal computer 82 shown in the figure, and video behavior recognition is performed on the video stream provided by the caller to identify the behavior type in the video stream. In the application scenario of the live broadcast platform, this server can also be used as a background monitoring service when running. For example, it can be used to perform video behavior analysis on the video stream received by the media server in the live broadcast platform, realize the monitoring of the video stream content, timely identify bad content, and notify the operation and maintenance interface of the live broadcast platform to escort the healthy development of network content.

[0068] On the basis of understanding the relevant application environment and development environment related to the technical solution of the present application disclosed above, please refer to the following description to continue to understand the various implementation methods of the present application. It should be pointed out that those skilled in the art should be aware of this: although the various methods of the present application are described based on the same concept and show commonality with each other, unless otherwise specified, these methods can be executed independently. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept. Therefore, concepts with the same expression, and concepts that are appropriately transformed only for convenience despite different conceptual expressions, should be understood equally.

[0069] The solution implemented in this application is applicable Figure 2 The artificial neural network principle framework shown in the figure can significantly improve the accuracy of temporal behavior positioning without sacrificing speed compared to the mainstream baseline algorithm R-C3D. This framework is based on the classic R-C3D two-stage framework and mainly consists of four parts:

[0070] (a) Video feature extraction network: The five convolutional layers from conv1a to conv5b in C3D are used as feature extractors.

[0071] (b) Temporal Candidate Network: It consists of two sub-networks: Temporal Locality-Aware Network (TLAN) and Temporal Proposal Suggestion Network (TPSN).

[0072] (c) Temporal Proposal Decision Module (TPDM): It is used to perform non-maximum suppression on the generated temporal candidate boxes, sort them from high to low according to confidence, and select the top N temporal candidate boxes.

[0073] (d) Temporal Action Classification Network (TACN): It includes a sub-action encoding module responsible for implementing pooling and a temporal action classification module responsible for implementing full connection. The two modules work together to encode the input temporal candidate boxes and then send them to the fully connected layer for classification and boundary refinement.

[0074] In this application, the four-tuple [T, H, W, C] is used to represent a three-dimensional convolution layer, where T, H, W represent the three-dimensional convolution kernel size (time, height, width), and C represents the number of filters (number of output channels). The symbol x1×x2×x3×x4 is used to represent the feature map size.

[0075] The artificial neural network architecture adopted in the present application is first given above, and the subsequent description is based on the architecture to help better understand the technical solution of the present application.

[0076] Please combine Figure 2 and Figure 3 A video behavior detection method of the present application is mainly implemented by an artificial neural network implemented according to the method, and includes the following steps:

[0077] Step S11, extracting features from the video stream to obtain a space-time feature map containing spatial and temporal feature information:

[0078] This step can be implemented by a feature extraction network in a pre-built artificial neural network.

[0079] Taking the live broadcast scenario as an example, when the video stream generated by the live broadcast user is uploaded to the media server deployed by the live broadcast platform, the method can be used to process the video stream. Specifically, the multiple convolutional layers of the C3D network can be used to gradually downsample the video stream to reduce the spatial resolution while keeping the temporal resolution unchanged, and obtain the spatiotemporal feature map:

[0080] Similar to R-C3D, multiple convolutional layers of the C3D network, generally five convolutional layers, are used to convolve the video stream layer by layer. While keeping the temporal resolution unchanged, each convolutional layer sequentially downsamples the video stream layer by layer, compresses the spatial resolution of the video stream step by step, and finally outputs the spatiotemporal feature map of the video stream after the fifth convolutional layer conv5b. The spatiotemporal feature map will be shared and called later. Specifically, the temporal receptive field is expanded through three layers of three-dimensional convolutional layers with a convolution kernel of 3×3×3, and the spatial resolution is reduced layer by layer while keeping the temporal resolution unchanged. The spatiotemporal feature map is finally output

[0081] Step S12, pooling the space-time feature map into a one-dimensional feature map that only retains the timing features of the video stream, evaluating the first probability score that the control area of ​​each timing point in the one-dimensional feature map belongs to the target timing behavior, and in parallel, evaluating the offset of a series of candidate boxes in the timing points of the one-dimensional feature map and the second probability score that the candidate box contains the target timing behavior:

[0082] This step can be implemented by a temporal candidate network pre-constructed in an artificial neural network. This step first obtains the one-dimensional feature map, and then performs two evaluation steps in parallel based on the one-dimensional feature map.

[0083] In the step of pooling the spatiotemporal feature map into a one-dimensional feature map that only retains the temporal features of the video stream, in order to obtain a one-dimensional feature map that retains the temporal features of the video stream based on the spatiotemporal feature map, a small three-dimensional convolutional network is added to achieve this. The spatial resolution of the spatiotemporal feature map is downsampled to 1×1 by 3D pooling to generate the one-dimensional feature map C pgn / pool This approach avoids the loss of spatial information that may be caused by large-scale average pooling in the baseline algorithm by downsampling the spatial resolution at multiple levels. The one-dimensional feature map obtained in this way can be used for further behavior temporal position perception and candidate box proposal.

[0084] See also Figure 4 The step of evaluating the first probability score that the control area of ​​each time point in the one-dimensional feature map belongs to the target time sequence behavior can be implemented by a temporal location awareness module (TLAN) pre-constructed in the temporal candidate network in the artificial neural network, and the step includes:

[0085] Step S1211: Slide a time sequence window of a preset length on the one-dimensional feature map to obtain the features of each time sequence point and its context information:

[0086] In the temporal location awareness module (TLAN), a temporal window of length l is first used to map the one-dimensional feature map C pgn / poolSlide up, so that each time point can obtain enough context information. The length of the time window can be preset. In one embodiment of the present application, the time window l is set to 3. Convolution is performed according to this length, and each time window is first mapped to a 512-d feature. For this reason, this embodiment uses a three-dimensional convolution layer to simulate this process ([3,1,1,512]).

[0087] Step S1212: fully connect the features corresponding to the time sequence window to the preset foreground and background binary classification network to determine the first probability score that the corresponding time sequence point control area belongs to the target time sequence behavior:

[0088] The 512-d features under the time window of length l are fully connected to a foreground / background binary classification network for category-independent foreground / background binary classification. The binary classification network is a fully connected layer, which is used to calculate the probability that the control area corresponding to the time point belongs to the target time sequence behavior, as the first probability score (also called action score: Actionness Score), to achieve the foreground / background binary classification of the time point. Similarly, this process is also simulated by a three-dimensional convolutional layer ([1,1,1,2]).

[0089] See also Figure 5 The step of evaluating the offset of a series of candidate boxes in the time points of the one-dimensional feature graph and the second probability score of the candidate box containing the target time sequence behavior can be implemented by a temporal candidate box suggestion module pre-constructed in the temporal candidate network in the artificial neural network, and the step includes:

[0090] Step S1221: Slide a time sequence window of a preset length on the one-dimensional feature map to obtain the features of each time sequence point and its context information:

[0091] Similar to the implementation of the temporal position perception module, in this step, the temporal window length is set to 3, and a three-dimensional convolutional layer ([3, 1, 1, 512]) is first used to encode context information at each temporal point of the one-dimensional feature map.

[0092] Step S1222: fully connect the features corresponding to the time series window to a preset two-classification network that is independent of the behavior category, and determine the offset of the candidate box corresponding to the corresponding time series point control area and the second probability score that the candidate box contains the target time series behavior:

[0093] Then, the features corresponding to each time point are used to predict the offset of a series of candidate boxes and the binary classification scores that are independent of the candidate box category as the second probability score, which represents the probability that the candidate box contains the target time behavior. This process can be simulated using two three-dimensional convolutional layers ([1,1,1,2k], [1,1,1,2k]). k represents the number of candidate boxes with preset values ​​at each time point.

[0094] As can be seen from the above, there are two types of steps in step S12 that run in parallel with each other, which are used to evaluate the first probability score and the second probability score respectively. In the artificial neural network, by adding a temporal position perception module, the step of estimating the first probability score is implemented, which is implemented by the temporal position perception module and the temporal candidate box suggestion module respectively. Among them, the temporal position perception module (TLAN) inputs the feature map And output a series of action scores (Actionness Score, indicating the probability that the control area of ​​the time point on the feature map belongs to the target time behavior); the temporal candidate box suggestion module (TPSN) inputs the same feature map and outputs a series of temporal behavior candidate boxes. The temporal location awareness module (TLAN) and the temporal candidate box suggestion module (TPSN) are designed as twin structures, implementing a multi-task learning mechanism in the artificial neural network, and jointly fully supervised learning the time segment scene classification task and the time regression learning task, which helps the network extract more discriminative features.

[0095] Step S13: Selecting several candidate boxes with higher confidence levels according to the first probability score and the second probability score:

[0096] The design of the aforementioned temporal position perception module can not only effectively improve the classification ability of temporal regions through multi-task learning and suppress the interference of background temporal fragments, but also its output probability score (category-independent binary classification score, actionness score) can also be used to generate more accurate temporal candidate frames. Specifically, the first probability score generated by the temporal position perception module can be used to revise the second probability score corresponding to the temporal candidate frame generated by the temporal candidate frame suggestion module, and then re-sort and select the first N temporal candidate frames.

[0097] In the preferred embodiment, please refer to Figure 6 , can be performed through the following specific steps:

[0098] Step S131: Count the first probability scores corresponding to the candidate boxes to obtain the first probability total scores of the candidate boxes:

[0099] There is a temporal correspondence between the temporal candidate boxes generated by the temporal behavior suggestion module and the first probability scores generated by the temporal position awareness module. Therefore, the first probability scores corresponding to each candidate box can be summarized.

[0100] Specifically, given a temporal candidate box p i =[s i ,e i ] and the category-independent behavior scores output by TLAN The confidence of the candidate boxes in the time series is calculated according to the behavior score to obtain the first probability total score of each candidate box:

[0101]

[0102] Step S132: perform weighted averaging of the first probability total scores and the second probability scores corresponding to each candidate frame system to obtain a weighted total score.

[0103] Since the temporal candidate frame module generates the second probability total score corresponding to each candidate frame, this step performs a weighted average of the first probability total score and the second probability score of the original TPSN temporal candidate frame. Specifically, assuming that the score of the candidate frame i obtained by the temporal candidate suggestion module TPSN is score(p i ). Then the newly generated weighted total score is:

[0104] p′ i =α*score(p i )+(1-α)*action_score(p i )

[0105] Step S133: sort the weighted total scores and select several candidate boxes with higher confidence:

[0106] On the basis of obtaining the weighted total score of each candidate box, the candidate boxes are sorted according to the weighted total score, and finally the first N candidate boxes with higher confidence are selected as the temporal candidate boxes generated in this stage for full connection operation. Here, N can be set as needed by those skilled in the art, for example, in this embodiment, it is set to 100, indicating that the first 100 candidate boxes are selected from the sorting results for subsequent processing.

[0107] When selecting a candidate box with a higher confidence, non-maximum suppression (NMS) is performed on the candidate box to select a relatively high-quality candidate box. The threshold IoU of NMS can be set by a person skilled in the art as needed, and a candidate box with a weighted total score higher than the threshold can be confirmed as a high-quality candidate box.

[0108] It can be seen that the temporal position of the target behavior is perceived by the temporal position perception module to form the first probability score, which is further summarized to obtain the first probability total score of each candidate box, and then the first probability total score and the second probability score corresponding to each candidate box are weighted and summarized, and the weighted total score is sorted to finally select the candidate box with a higher confidence level. In this process, the output of the added temporal position perception module is utilized to correct the score of the candidate box generated by the temporal candidate box suggestion module, which can improve the accuracy of the candidate box recommended by the artificial neural network.

[0109] Step S14: Using the space-time feature map as a shared feature, the candidate frames are pooled into multiple feature maps with the same dimension and output to the fully connected layer, which completes the video behavior classification and regression and outputs the classification result data of the video stream:

[0110] As mentioned above, the spatiotemporal feature map output after C3D multi-layer convolution is used as a shared feature and is shared and called in this step. Similar to the R-C3D algorithm, the features are extracted from the spatiotemporal feature map according to the candidate boxes selected in the previous step for pooling, and then the pooled feature maps corresponding to each candidate box are fully connected to complete the classification and recognition of video behaviors and boundary pruning. This step can be implemented by the behavior recognition network built into the artificial neural network.

[0111] In one embodiment, the R-C3D algorithm principle can be followed to apply RoI (Region of Interest) pooling to share the spatial-temporal feature map. The feature of is taken as input, and it is divided into multiple space-time sub-feature maps. Then, the result feature map is extracted by maximum pooling in each space-time sub-feature map. The result feature map is finally output to the fully connected layer for video behavior classification and regression processing to obtain the corresponding start time and result time of the video behavior. Assume that C3D outputs a feature map of size 512xL / 8x7x7, and assume that the length (sequence length) of one candidate box is t s , then the size of this candidate box is 512xt s x7x7, a dynamically sized pooling kernel is used here Finally, a feature map of size 512x1x4x4 is obtained. Please note that when processing with traditional RoI pooling, the division of the time domain is limited to one unit, such as Figure 7 (a) As shown in the 3D RoI Pooling part, the final feature map is of size 1x4x4. Although the candidate boxes selected in this application can also be used, unfortunately, the timing information is lost here, resulting in insufficient basis for classification decisions in the subsequent fully connected layer.

[0112] In another embodiment, an improvement is made on the basis of RoI Pooling, referred to as SSTP encoding, which aims to provide richer timing features for the fully connected layer. The implementation process can be defined as the following expression:

[0113] vf SSTP =max_pooling(S(F(p i )))

[0114] Among them, given a temporal candidate box p i =[s i ,e i ], F(·) function means to extract p from the feature map i Corresponding features The S(·) function represents the division into equal-sized In this embodiment, t s It is set to a unit greater than 1, preferably 4, to retain more timing feature information through this time domain parameter.

[0115] See also Figure 8 To implement the solution expressed by the formula, in this embodiment, step S14 includes the following specific steps:

[0116] Step S1411: extract the corresponding feature map to be pooled from the space-time feature map according to the selected candidate frames:

[0117] In this step, the temporal candidate frame with higher confidence selected in the previous step corresponds to the temporal feature map Extract the corresponding feature map to be pooled.

[0118] Step S1412: divide the feature map to be pooled into multiple space-time sub-feature maps, and retain the temporal features corresponding to the candidate box between the multiple space-time sub-features:

[0119] The difference from the prior art is that the parameter t s It is set to 4. Therefore, when the spatial-temporal feature map corresponding to the candidate box is divided into multiple spatial-temporal sub-feature maps, the application retains the time series length and enriches the feature information in the time series dimension by expanding the time series to multiple units.

[0120] Step S1413: Apply the maximum pooling function to pool the space-time sub-feature map to obtain a result feature map:

[0121] On the basis of obtaining the spatial-temporal sub-feature map, a maximum pool is applied to map the relevant maximum values ​​into a result feature map of uniform size, and the following is obtained: Figure 7 Part (b) shows the 4x2x2 feature map.

[0122] Step S1414: output the result feature map obtained by pooling to the fully connected layer:

[0123] The resulting feature map obtained by pooling is output to two fully connected layers, including a classification fully connected layer and a regression fully connected layer. The classification fully connected layer scores the video behavior in each category according to the resulting feature map to achieve classification, and the regression fully connected layer performs boundary trimming according to the resulting feature map to achieve border regression and obtain the corresponding start time and end time of the video behavior, thereby completing video behavior recognition.

[0124] In another embodiment, an improvement is made on the basis of the previous embodiment, which is referred to as the CSSTP encoding method. Specifically, the candidate frame encoding context information can be further extended on the basis of the extended temporal features. The implementation process can be defined as the following expression:

[0125] vf CSSTP =max_pooling(S(F(E(p i ))))

[0126] Among them, given a temporal candidate box p i =[s i ,e i ], temporal candidate box p i The duration can be written as d i =e i -s i The function E(·) represents the context feature encoding. Specifically, each temporal candidate box p i Expand to p′ i =[s′ i ,e′ i ], where s′ i =s i -d i / 2, e′ i =e i +d i / 2. Then, p′ can be i Divided into three consecutive segments: and Then, the eigenvectors of the three parts are calculated independently and Then connect them together. Specifically, in this embodiment, the feature map of each part of each temporal candidate box Can be divided into In this embodiment, the height (H) and width (W) of the video frame are both 112, so the feature map input to the classification network is and is divided into 1×2×2 sub-grids, It is divided into 2×2×2 sub-grids.

[0127] See also Fig. 9 To implement the solution expressed by the formula, in this embodiment, step S14 includes the following specific steps:

[0128] Step S1421: extract the feature maps to be pooled that belong to the continuous segments before and after the selected candidate frames from the space-time feature map according to the time sequence length occupied by each candidate frame:

[0129] As expressed in the formula, in addition to obtaining the to-be-pooled feature map of the segment of the candidate box itself, the segments before and after the segment are also obtained. The lengths of these two extended segments generally do not exceed half of the total time occupied by the candidate box. The pooled feature map obtained in this way can further enrich the feature information of the video content.

[0130] Step S1422: split the to-be-pooled feature maps of each segment into space-time sub-feature maps, and retain the temporal features corresponding to the candidate box between multiple space-time features:

[0131] At the same time, for each segment, its feature map to be pooled is divided into multiple spatial and temporal sub-feature maps, and due to the parameter t s is greater than 1, and is set to 4 as mentioned above. Therefore, the temporal span is also expanded, so that the spatial-temporal sub-feature map of each candidate box is enriched not only with contextual information but also with temporal feature information.

[0132] Step S1423: Apply the maximum pooling function to pool the spatial-temporal sub-feature maps of each segment and unify them into a result feature map:

[0133] Similarly, after obtaining the space-time sub-feature maps of each segment, the result feature map is obtained by maximum pooling, such as Figure 7 As shown in part (c), in the feature map obtained in this embodiment, the candidate box's own fragments are pooled into a size of 2x2x2, and the fragments before and after it are pooled into a size of 1x2x2, and finally these images are connected to a unified size of 4x2x2.

[0134] Step S1424: Connect the result feature maps obtained by pooling and output them to the fully connected layer:

[0135] Similar to the above, the final output feature map is output to two fully connected layers for classification and regression to complete video behavior recognition.

[0136] In another embodiment, an improvement is made based on step S1423 of the previous embodiment, referred to as SCE encoding, and correlation encoding is performed on each space-time sub-feature graph obtained in this step, including the space-time sub-feature graphs of each segment and the space-time sub-feature graphs generated by connecting these segments, such as Fig.10 As shown, the following steps are added:

[0137] Step S1423', based on the space-time sub-feature graphs and result feature graphs of each fragment, on the basis of retaining their structured coding features, perform correlation calculation on the features at the same spatial point along the temporal dimension to obtain corresponding coding features for output to the fully connected layer.

[0138] Specifically, step S1423′ is to organize the structural relationship between the spatial-temporal sub-feature graphs of the same spatial point in each segment and the result feature graph, and express the correlation between the graphs by encoding so that they can be subsequently passed to the fully connected layer for reference. The principle can be found in Figure 7 Part (d) shows:

[0139] Assume that the temporal features of the four different components (i.e., the spatial-temporal sub-feature graphs of each fragment and the result feature graph unified from them) obtained after CSSTP are expressed as:

[0140]

[0141] On the basis of retaining these structured coding features, the correlation of the features at the same spatial point along the time series dimension is calculated here. The calculation formula is as follows:

[0142]

[0143] in represents the re-encoded features, w i represents the weight parameter to be learned, f i Represents the component features after the temporal behavior is segmented and pooled. Naturally, this linear function can be implemented using a 3D convolutional network with a convolutional layer size of ([4, 1, 1, 512]).

[0144] In step S14, the sub-behavior encoding module may implement any of the aforementioned embodiments to implement encoding through pooling, and the temporal behavior classification module may be responsible for classifying and regressing the encoded features.

[0145] In response to dynamic structural challenges, the above-listed embodiments basically follow four principles: context coding, structured coding, inter-atomic temporal correlation coding, and end-to-end joint debugging. These principles not only involve the specific operations of pooling, but also include comprehensive considerations of pooling methods and network design. It can be seen that in step S14, starting from the regional and instance-level feature coding, on the basis of traditional RoI pooling, this application also explores different pooling methods and feature coding methods, and improves the accuracy of temporal behavior classification to varying degrees through different embodiments.

[0146] When the method of the present application is implemented with an artificial neural network, the following scheme can be designed to design the relevant cost function (or loss function, objective function):

[0147] This application implements an end-to-end two-stage positioning framework, in which three sub-networks play a key role: the temporal candidate box suggestion module (TPSN), the temporal location awareness module (TLAN), and the temporal behavior classification module (TACN). In particular, for the temporal behavior classification module (TACN), the cross entropy loss function and the smooth L1 loss function are used for classification tasks and regression tasks, respectively. Among them, the loss function of TACN is defined as follows:

[0148]

[0149] The first half of the formula represents the classification loss, and the second half represents the regression loss. λ represents the balance parameter, which is set to 1 in the experiment. i represents the index of the temporal behavior candidate box, p i Indicates the predicted probability that the temporal behavior candidate box i belongs to a certain category of temporal behavior (the number of categories here includes the background category), Represents the supervision information of the temporal behavior candidate box i. Represents the relative offset of the predicted temporal behavior candidate box i, Represents the actual offset between the temporal behavior candidate box i and the target temporal behavior.

[0150] Similarly, formula (4.2) also applies to the temporal candidate box suggestion module (TPSN). The difference is It represents the probability that the anchor box i belongs to the target temporal behavior (category-independent foreground / background binary classification). During the training phase, if the candidate box (anchor box) belongs to a positive sample, it is set to 1, otherwise it is set to 0. The corresponding p i Represents the prediction score of anchor box i. Represents the relative offset between the predicted anchor box i and the annotated true temporal behavior.

[0151] The temporal location-aware module (TLAN) is essentially a classifier, so the cross entropy loss function is also applicable. Specifically, the objective function of TLAN is as follows:

[0152]

[0153] Where N represents all temporal positions. In the experiment, since the number of input video frames is L and the temporal downsampling step is 8, N is L / 8. i Represents the probability that the predicted i-th temporal position control area is within the target behavior (category-independent foreground / background binary classification). is the supervision information of the i-th time point. During the training phase, if the time point i is a positive sample, it is set to 1, if it is a negative sample, it is set to 0, otherwise it is set to -1. Therefore, the end-to-end network loss function proposed here is defined as follows:

[0154] L=αL TPSN +βL TLAN +γL TACN (4.4)

[0155] Among them, α, β and γ are all equilibrium parameters, which are all set to 1 in the experiment here.

[0156] In order to demonstrate the effectiveness of the artificial neural network structure implemented by the technical solution of the present application, experiments were conducted using the standard data set THUMOS14 based on the artificial neural network implemented by the solution of the present application. The experimental results are briefly reported below:

[0157] First, see Fig.11 By comparing the recall rate-average temporal candidate box statistics of the R-C3D related method and the method of the present application under the same tIoU conditions, it is found that: when the same number of temporal candidates is taken, the recall rate of the temporal candidate box generated by the technical solution of the present application is significantly higher than that of R-C3D by nearly 3%-5%.

[0158] Secondly, the comparison chart of recall rate-tIoU between R-C3D and this application with the same number of candidate boxes is as follows Fig.12 As shown: Under the same recall rate condition, the temporal candidate box tIoU generated by the method of the present application is significantly higher by 3%-7%, indicating that the temporal candidate box generated by the method of the present application is more accurate at the boundary.

[0159] Finally, the performance improvement of this method compared with the R-C3D algorithm is statistically analyzed using mAP as the evaluation indicator as shown in the following table. It can be seen that all indicators are better than the method based on the traditional R-C3D implementation:

[0160]

[0161] In order to demonstrate the effectiveness of the encoding method implemented by the technical solution of the present application, an experiment was conducted using the standard data set THUMOS14 based on the artificial neural network implemented by the solution of the present application. The experimental results are briefly reported below:

[0162] First, we directly compare the mAP reports of R-C3D and this application on the THUMOS14 dataset. It is worth noting that this experiment only replaces the pooling method in R-C3D, that is, it only replaces the content of step S14 on the basis of the original algorithm of R-C3D, and does not directly modify the temporal candidate box in TPSN. Therefore, this experiment can effectively prove the improvement of the temporal behavior classification capability of step S14 of this application. As shown in the following table:

[0163]

[0164] As can be seen from the table above, the mAP of the solution adopted by this application is far higher than that of R-C3D under all tIoU conditions by 8%-10%. In particular, after adding the corresponding embodiment of sub-behavior correlation coding, the performance is significantly improved compared with the embodiment without correlation coding. This experiment fully demonstrates the effectiveness of this application.

[0165] Secondly, the present application further proves that its method does not have too much positive impact on the generation of temporal candidate boxes. As shown in the following table:

[0166]

[0167] The above table shows that when step S14 (SCE-NET) of the present application is independently combined with R-C3D, the average recall rate is less than 0.4%-1% under different numbers of temporal candidate boxes. Therefore, these two experiments fully demonstrate that step S14 of the present application improves the video temporal behavior detection capability (mAP) by improving the classification ability of the network.

[0168] See also Fig.13 As shown, in order to fully utilize the artificial neural network implemented by the present application, a video behavior response method is proposed, which includes the following steps:

[0169] Step S21: Obtain the video stream of the live broadcast user:

[0170] In the live broadcast platform, live broadcast users of all parties upload their own video streams to the media server one after another. After these video streams are received by the media server, they can be submitted to the artificial neural network implemented in this application for video behavior detection. Specifically, the media server can call the interface provided by the artificial neural network for detection, or a monitoring process can actively collect the video stream for detection.

[0171] Step S22: import the video stream into an artificial neural network to implement video behavior recognition. The artificial neural network executes the video behavior detection method and generates classification result data of the video stream accordingly:

[0172] After obtaining the video stream, the corresponding video stream can be input into the artificial neural network to implement video behavior recognition. The artificial neural network executes the video behavior detection method described in this application, detects the behavior implemented by the live user from the video stream, and then determines the corresponding classification result data, mainly including the type related to the video behavior and its start time and end time, so as to determine the video segment to which the relevant video behavior belongs through this classification result data.

[0173] Step S23: Output a preset and corresponding notification message to the live broadcast room user according to the classification result data output by the artificial neural network:

[0174] After identifying the corresponding video behavior or even obtaining the corresponding video segment from the video stream based on the classification result data fed back by the artificial neural network, corresponding control can be made according to the specific video behavior based on the preset logic. For example, a corresponding notification message can be generated and fed back to the operation and maintenance interface of the live broadcast platform or the live broadcast users of the video stream, etc., so as to facilitate the relevant parties to further control and supervise the video stream.

[0175] It can be understood that by applying the video behavior detection method of the present application, specific video behaviors can be more accurately and timely identified from real-time video streams, which can achieve twice the result with half the effort in improving the intelligence level of the network platform.

[0176] Furthermore, by functionalizing the various embodiments of the above-mentioned video behavior detection method, a neural network implementation device of the present application can be constructed. According to this idea, please refer to Figure 2 In a typical embodiment, the device comprises:

[0177] A neural network implementation device provided to meet one of the purposes of the present application includes:

[0178] Feature extraction network a, used to extract features from the video stream to obtain a spatiotemporal feature map containing spatial and temporal feature information;

[0179] A temporal candidate network b, which pools the spatiotemporal feature map into a one-dimensional feature map that only retains the temporal features of the video stream, and includes a temporal position perception module TLAN and a temporal candidate box suggestion module TPSN, wherein the temporal position perception module TLAN is used to evaluate the first probability score that the control area of ​​each temporal point in the one-dimensional feature map belongs to the target temporal behavior; the temporal candidate box suggestion module TPSN runs in parallel with the temporal position perception module and is used to evaluate the offset of a series of candidate boxes in the temporal points of the one-dimensional feature map and the second probability score that the candidate box contains the target temporal behavior;

[0180] A temporal candidate box decision network c, used to select a plurality of candidate boxes with higher confidence according to the first probability score and the second probability score;

[0181] The behavior recognition network d is used to use the spatiotemporal feature map as a shared feature, pool the candidate boxes into multiple feature maps with the same dimension and output them to the fully connected layer, which completes the video behavior classification and regression and outputs the classification result data of the video stream.

[0182] In order to facilitate the execution of the present application, an electronic device is provided, including a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of a video behavior detection / response method as described above.

[0183] It can be seen that the memory is suitable for using a non-volatile storage medium. By implementing the aforementioned method as a computer program and installing it in an electronic device such as a mobile phone, the relevant program code and data are stored in the non-volatile storage medium of the electronic device. The program is further run by the central processor of the electronic device, and it is transferred from the non-volatile storage medium to the memory for execution, so that the desired purpose of the present application can be achieved. Therefore, it can be understood that in one embodiment of the present application, a non-volatile storage medium can also be provided, which stores a computer program implemented according to various embodiments of the video behavior detection / response method in the form of computer-readable instructions. When the computer program is called and executed by the computer, the steps included in the method are executed.

[0184] In summary, the present application can improve the accuracy of artificial neural networks in identifying video behaviors, significantly improve the effect of online video behavior detection, and is widely applicable to various Internet application scenarios that require online video behavior detection.

[0185] It will be appreciated by those skilled in the art that the present application package relates to devices for performing one or more of the operations and methods described in the present application. These devices may be specially designed and manufactured for the desired purpose, or may also include known devices in general-purpose computers. These devices have computer programs stored in their memory, which are selectively activated or reconstructed. Such computer programs may be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, respectively, the computer readable medium including but not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable medium includes any medium that stores or transmits information in a readable form by a device (e.g., computer).

[0186] Those skilled in the art will appreciate that each block in these structure diagrams and / or block diagrams and / or flow charts and combinations of blocks in these structure diagrams and / or block diagrams and / or flow charts can be implemented using computer program instructions. Those skilled in the art will appreciate that these computer program instructions can be provided to a general-purpose computer, a professional computer, or a processor of other programmable data processing methods to implement, thereby executing the scheme specified in the block or multiple blocks of the structure diagrams and / or block diagrams and / or flow charts disclosed in this application through the processor of the computer or other programmable data processing method.

[0187] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0188] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video behavior response method, characterized in that: Applied to the media server, including the following steps: Acquire a video stream of a live broadcast user, the video stream being uploaded to the media server by the live broadcast user; Importing the video stream into an artificial neural network to implement video behavior recognition, the artificial neural network executes the video behavior detection method and generates classification result data of the video stream accordingly, including the type of video behavior related to the type and its start time and end time, so as to determine the video segment to which the relevant video behavior belongs through the classification result data; According to the classification result data output by the artificial neural network, after identifying the corresponding video behavior and the corresponding video segment from the video stream according to the classification result data, according to the preset logic, make corresponding control according to the specific video behavior, including outputting a preset and corresponding notification message to the live broadcast room user; Wherein, the video behavior detection method comprises: Extract features from the video stream generated by the live broadcast user to obtain a spatiotemporal feature map containing spatial and temporal feature information; Pooling the spatiotemporal feature map into a one-dimensional feature map that only retains the temporal features of the video stream, evaluating a first probability score that the control area of ​​each temporal point in the one-dimensional feature map belongs to the target temporal behavior, and in parallel, evaluating the offset of a series of candidate boxes in the temporal points of the one-dimensional feature map and a second probability score that the candidate box contains the target temporal behavior; Selecting a plurality of candidate boxes with higher confidence levels according to the first probability score and the second probability score; Using the space-time feature map as a shared feature, the candidate frames are pooled into multiple feature maps with the same dimension and output to a fully connected layer, the fully connected layer completes video behavior classification and regression, and outputs classification result data of the video stream, the classification result data includes the type related to the video behavior and its start time and end time, and the video segment to which the related video behavior belongs is determined through the classification result data, and the live broadcast user is notified; The step of pooling the candidate boxes into multiple feature maps with the same dimension and outputting them to the fully connected layer includes: extracting the feature maps to be pooled belonging to continuous segments before and after the selected candidate boxes from the space-time feature map according to the time length occupied by each selected candidate box, the segment length before the space-time feature map and the segment length after the space-time feature map shall not exceed half of the total time length occupied by the candidate boxes; dividing the feature maps to be pooled of each segment into space-time sub-feature maps, and retaining the time features corresponding to the candidate box between multiple space-time sub-feature maps; applying the maximum pooling function to pool the space-time sub-feature maps of each segment respectively and unifying them into a result feature map; based on the space-time sub-feature maps of each segment and the result feature map, while retaining their structured coding features, performing correlation calculation on the features at the same spatial point along the time dimension to obtain corresponding coding features for output to the fully connected layer.

2. The method according to claim 1, characterized in that: In the step of pooling the spatiotemporal feature map into a one-dimensional feature map that only retains the temporal features of the video stream, the spatial resolution of the spatiotemporal feature map is downsampled to a size of 1x1 through 3D pooling to obtain the one-dimensional feature map.

3. According to the method of claim 1, the step of evaluating the first probability score that the control area of ​​each time sequence point in the one-dimensional feature graph belongs to the target time sequence behavior comprises: Slide a time sequence window of a preset length on the one-dimensional feature map to obtain the features of each time sequence point and its context information; The features corresponding to the time sequence window are fully connected to the preset foreground and background binary classification network to determine the first probability score that the corresponding time sequence point control area belongs to the target time sequence behavior.

4. According to the method of claim 1, the step of evaluating the offset of a series of candidate boxes in the time points of the one-dimensional feature map and the second probability score that the candidate boxes contain the target time sequence behavior comprises: Slide a time sequence window of a preset length on the one-dimensional feature map to obtain the features of each time sequence point and its context information; The features corresponding to the timing window are fully connected to a preset two-classification network that is independent of the behavior category, and the offset of the candidate box corresponding to the corresponding timing point control area and the second probability score that the candidate box contains the target timing behavior are determined.

5. The method according to claim 1, characterized in that The step of selecting a plurality of candidate boxes with higher confidence according to the probability score comprises: Counting the first probability scores corresponding to the candidate boxes to obtain the first probability total scores of the candidate boxes; Taking a weighted average of the first probability total score and the second probability score corresponding to each candidate frame system to obtain a weighted total score; The weighted total scores are sorted to select several candidate boxes with higher confidence.

6. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

7. A non-volatile storage medium, characterized in that: The computer program implemented according to the method described in any one of claims 1 to 5 is stored in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

Images