An AI-powered method for detecting anomalies in audio and video broadcast content

Abstract

This invention provides an artificial intelligence method for detecting anomalies in audio and video broadcast content, relating to the field of video processing technology. The method includes: decoding the video to be processed to obtain multiple video frames, and performing feature extraction processing to obtain a spatiotemporal feature map; obtaining partitioned spatiotemporal feature vectors based on the spatiotemporal feature map; inputting the partitioned spatiotemporal feature vectors into an encoding model to obtain comprehensive spatiotemporal feature information; and inputting the comprehensive spatiotemporal feature information into a classification and discrimination model to obtain anomaly type identification results. According to this invention, feature extraction processing at multiple levels and in multiple partitions can effectively capture local and global information in the video, facilitating accurate understanding of the video content. This enables the identification of anomaly types in consecutive multi-frame videos and is applicable to a wider range of scenarios and anomaly types, improving recognition accuracy and applicability.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to an artificial intelligence method for detecting anomalies in audio and video broadcast content. Background Technology

[0002] In related technologies, simple image feature extraction methods are typically used to detect anomalies in broadcast television videos. These methods usually only identify a few specific, simple scenes, such as black screens or still frames. Specific identification methods typically include calculating image brightness (below a threshold is considered a black screen) and extracting graphic features from adjacent frames to calculate similarity (multiple similar frames are considered still frames). Therefore, the methods for identifying anomalies in related technologies can only identify a limited number of anomalies, cover relatively simple scenes, and lack an understanding of the video content, resulting in a high false positive rate.

[0003] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides an artificial intelligence identification method for detecting anomalies in audio and video broadcast content, which can solve the technical problems of limited types of anomalies, limited scenarios, and high false recognition rate in related technologies.

[0005] According to a first aspect of the present invention, a method for artificial intelligence identification of anomalies in audio and video broadcast content is provided, comprising:

[0006] The video to be processed is decoded to obtain multiple video frames to be processed;

[0007] Feature extraction is performed on a first preset number of consecutive video frames to be processed to obtain spatiotemporal feature maps of multiple feature channels;

[0008] Based on the spatiotemporal feature map, obtain the partition spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map;

[0009] The spatiotemporal feature vectors of multiple partitions are input into the encoding model for processing to obtain comprehensive spatiotemporal feature information.

[0010] The comprehensive spatiotemporal feature information is input into the classification and discrimination model for processing to obtain the anomaly type identification result.

[0011] According to the present invention, feature extraction processing is performed on a first predetermined number of consecutive video frames to be processed to obtain spatiotemporal feature maps of multiple feature channels, including:

[0012] By using multiple 3D convolutional layers, a layer-by-layer feature extraction process is performed on a first preset number of consecutive video frames to be processed to obtain the first 3D feature map output by each 3D convolutional layer.

[0013] The first 3D feature map output from each 3D convolutional layer is fused layer by layer to obtain spatiotemporal feature maps with multiple feature channels.

[0014] According to the present invention, the first 3D feature map output from each 3D convolutional layer is subjected to layer-by-layer feature fusion to obtain a spatiotemporal feature map with multiple feature channels, including:

[0015] The first 3D feature map of each level is processed by a 3D convolution kernel with a kernel size of 1×1×1 to obtain a second preset number of second 3D feature maps of each level.

[0016] Based on the size and number of layers of the second 3D feature map at each level, a second preset number of second 3D feature maps at each level are recursively processed to obtain spatiotemporal feature maps with multiple feature channels, wherein the number of feature channels of the spatiotemporal feature map is the second preset number.

[0017] According to the present invention, based on the size and number of layers of the second 3D feature map at each level, a second preset number of second 3D feature maps at each level are recursively processed to obtain spatiotemporal feature maps with multiple feature channels, including:

[0018] When l = L, the second 3D feature map of the Lth layer is set as the third 3D feature map of the Lth layer, where L is the number of 3D convolutional layers;

[0019] Upsample the third 3D feature map of the Lth level to obtain a fourth 3D feature map with the same size as the second 3D feature map of the (L-1)th level.

[0020] The fourth 3D feature map of the Lth level and the second 3D feature map of the L-1th level with the same feature channels are fused to obtain the third 3D feature map of the L-1th level.

[0021] When l≠L, the third 3D feature map of the l-th level is upsampled to obtain a fourth 3D feature map with the same size as the second 3D feature map of the (l-1)-th level.

[0022] The fourth 3D feature map of the l-th level and the second 3D feature map of the (l-1)-th level with the same feature channels are fused to obtain the third 3D feature map of the (l-1)-th level.

[0023] When the size of the third 3D feature map reaches the preset size, the recursion stops, and the third 3D feature map of multiple feature channels is determined as the spatiotemporal feature map of multiple feature channels.

[0024] According to the present invention, obtaining partitioned spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map based on the spatiotemporal feature map includes:

[0025] The spatiotemporal feature map is divided along the temporal direction, width direction, and height direction to obtain multiple partition feature maps;

[0026] Flatten the partition feature map into partition feature vectors;

[0027] Based on the temporal direction index of the partition feature map in the spatiotemporal feature map, as well as the indexes in the width and height directions, determine the spatiotemporal position code corresponding to the partition feature map;

[0028] The spatiotemporal location code is fused with the partition feature vector to obtain the partition spatiotemporal feature vector.

[0029] According to the present invention, the spatiotemporal feature vectors corresponding to multiple partitions are input into an encoding model for processing to obtain comprehensive spatiotemporal feature information, including:

[0030] In each level of the encoding model, a multi-head attention mechanism is used to process the input feature vector to obtain the output feature information of each attention head. In the first level of the encoding model, the input feature vector is the partitioned spatiotemporal feature vector.

[0031] The output feature information of each attention head is concatenated to obtain the concatenated feature information;

[0032] The concatenated feature information is processed by a weight matrix to obtain an output feature vector, which is then used as the input feature vector for the next level.

[0033] After processing through multiple levels of the encoding model, the output feature vectors are combined to obtain comprehensive spatiotemporal feature information.

[0034] According to the present invention, the comprehensive spatiotemporal feature information is input into a classification and discrimination model for processing to obtain anomaly type identification results, including:

[0035] The comprehensive spatiotemporal feature information is input into the multi-layer perception layer of the classification and discrimination model for processing to obtain the anomaly type feature vector;

[0036] The feature vectors of the abnormal type are input into the activation layer of the classification and discrimination model for processing to obtain the abnormal type recognition result.

[0037] According to the present invention, the feature vector of the abnormal type is input into the activation layer of the classification and discrimination model for processing to obtain the abnormal type recognition result, including:

[0038] The feature vectors of the heterogeneous types are input into the activation layer of the classification and discrimination model for processing, and the probability data of various heterogeneous types are output.

[0039] The anomaly type corresponding to the maximum probability data is taken as the anomaly type identification result.

[0040] According to a second aspect of the present invention, a system for artificial intelligence identification of anomalies in audio and video broadcast content is provided, comprising:

[0041] The decoding module is used to decode the video to be processed and obtain multiple video frames to be processed.

[0042] The feature extraction module is used to perform feature extraction processing on a first preset number of consecutive video frames to be processed, and obtain spatiotemporal feature maps of multiple feature channels.

[0043] The partitioning module is used to obtain partitioned spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map based on the spatiotemporal feature map.

[0044] The encoding module is used to input the spatiotemporal feature vectors of multiple partitions into the encoding model for processing to obtain comprehensive spatiotemporal feature information;

[0045] The identification module is used to input the comprehensive spatiotemporal feature information into the classification and discrimination model for processing, and obtain the anomaly type identification result.

[0046] According to a third aspect of the present invention, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the method for artificial intelligence recognition of anomalies in audio and video broadcast content.

[0047] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0048] According to the present invention, feature extraction processing at multiple levels and in multiple partitions can effectively capture local and global information in a video, facilitating accurate understanding of the video content and enabling the identification of anomalies across multiple consecutive frames. This method is applicable to a wider range of scenarios and anomalies, improving both accuracy and applicability. Furthermore, feature maps of different scales can be obtained through multiple 3D convolutional layers, and these feature maps can be fused using a feature pyramid approach. This results in a spatiotemporal feature map containing rich local and global feature information, which is beneficial for understanding the video content and improving the accuracy of anomaly identification. The spatiotemporal feature map can also be partitioned to obtain partition feature vectors suitable for the Transformer model. Encoding based on the temporal and spatial positions of these partition feature vectors helps the model determine the location of each partition, leading to a more accurate understanding of the video content and further improving the accuracy of anomaly identification. Furthermore, a multi-head attention mechanism can be used to fuse spatiotemporal feature vectors from multiple channels and partitions, resulting in comprehensive spatiotemporal feature information that integrates multiple scales and local and global feature information. This allows for the acquisition of spatial and temporal dependencies between different input feature vectors, leading to a more accurate understanding of video content and improved accuracy in anomaly identification. Further, a classification model can be used to process the comprehensive spatiotemporal feature information to fully utilize the multi-scale, local, and global feature information, as well as spatial and temporal dependencies contained within it, thereby obtaining anomaly identification results. This improves the accuracy of anomaly identification and enhances the robustness and generalization ability of the classification.

[0049] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0051] Figure 1 An exemplary flowchart illustrates a method for artificial intelligence identification of anomalies in audio and video broadcast content according to an embodiment of the present invention;

[0052] Figure 2 An exemplary schematic diagram illustrates the application of an artificial intelligence identification method for detecting anomalies in audio and video broadcast content according to an embodiment of the present invention;

[0053] Figure 3 A block diagram of an artificial intelligence system for detecting anomalies in audio and video broadcast content according to an embodiment of the present invention is shown as an example. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0056] Figure 1 An exemplary flowchart illustrates a method for detecting anomalies in audio and video broadcast content using artificial intelligence, according to an embodiment of the present invention. The method includes:

[0057] Step S1: Decode the video to be processed to obtain multiple video frames to be processed;

[0058] Step S2: Perform feature extraction processing on a first preset number of consecutive video frames to be processed to obtain spatiotemporal feature maps of multiple feature channels;

[0059] Step S3: Based on the spatiotemporal feature map, obtain the partition spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map;

[0060] Step S4: Input the spatiotemporal feature vectors corresponding to multiple partitions into the encoding model for processing to obtain comprehensive spatiotemporal feature information;

[0061] Step S5: Input the comprehensive spatiotemporal feature information into the classification and discrimination model for processing to obtain the anomaly type identification result.

[0062] The artificial intelligence identification method for detecting anomalies in audio and video broadcast content according to embodiments of the present invention can effectively capture local and global information in the video through feature extraction processing at multiple levels and in multiple partitions, which helps to accurately understand the video content, thereby realizing the identification of anomaly types in multiple consecutive frames of the video, and can be applied to more scenarios and anomaly types, improving the accuracy and applicability of identification.

[0063] Example 1:

[0064] According to one embodiment of the present invention, in step S1, the video to be processed can be received via protocols such as UDP or RTSP, and the video to be processed can be decoded to obtain multiple video frames to be processed. That is, the video to be processed is composed of multiple video frames to be processed, each video frame to be processed can be obtained, and these video frames to be processed are continuous and can form a sequence of video frames to be processed.

[0065] According to an embodiment of the present invention, in step S2, when processing video frames to be processed, a first preset number of video frames to be processed can be processed at a time. Feature extraction processing can be performed on the consecutive first preset number of video frames to be processed through multiple 3D convolutional layers. When inputting multiple 3D convolutional layers, the consecutive first preset number of video frames to be processed can form a tensor with dimensions T×H×W×C0 for input, where T is the number of video frames, i.e., the first preset number, H is the height of each video frame to be processed, W is the width of each video frame to be processed, and C0 is the number of channels. For example, when inputting, each pixel in the video to be processed has data in three channels: R, G, and B. Therefore, C0 = 3.

[0066] According to one embodiment of the present invention, during processing, 3D convolutional layers can process the input tensor, and the dimensions of the output tensor may change compared to the input tensor. Furthermore, to obtain feature maps at multiple scales for a more comprehensive acquisition of global and local feature information, a 3D feature pyramid structure can be employed to fuse the output tensors of different 3D convolutional layers. This results in a final spatiotemporal feature map that contains richer local and global feature information, which is more conducive to understanding video content and improving the accuracy of anomaly type recognition results.

[0067] According to one embodiment of the present invention, feature extraction processing is performed on a first preset number of consecutive video frames to be processed to obtain spatiotemporal feature maps of multiple feature channels, including: performing layer-by-layer feature extraction processing on the first preset number of consecutive video frames to be processed through multiple 3D convolutional layers to obtain a first 3D feature map output by each 3D convolutional layer; and performing layer-by-layer feature fusion on the first 3D feature maps output by each 3D convolutional layer to obtain spatiotemporal feature maps of multiple feature channels.

[0068] According to one embodiment of the present invention, each 3D convolutional layer may have multiple 3D convolutional kernels, and the size of each 3D convolutional kernel is K. t ×K h ×K w K t =2,K h =3,K w =3. The parameters of the convolution kernel can be obtained through training.

[0069] According to an embodiment of the present invention, the 3D convolutional layer can process the input tensor of the layer by the following formula (1) to obtain the output tensor, wherein the input tensor is the output tensor of the previous 3D convolutional layer, and the input tensor of the first 3D convolutional layer is the aforementioned first preset number of video frames to be processed.

[0070]

[0071] in, To perform 3D convolution processing on the feature map of the m-th channel in the input tensor of the l-th 3D convolutional layer, the c-th convolutional kernel is located at the δ-th channel. t Frame, δ h row, δ w Column weight parameters, For the m-th channel of the input tensor of the l-th 3D convolutional layer, at the t·s-th position... t +δ t Frame i·s h +δ h row, j·s w +δ w The pixel values ​​of the columns of pixels, where s t s is the time step. h Let s be the step size in the height direction. w Let be the step size in the width direction, t be the number of steps in the time direction, i be the number of steps in the height direction, j be the number of steps in the width direction, and C be the step size in the width direction. l-1 The number of channels in the input tensor. Let be the bias of the c-th convolutional kernel in the l-th 3D convolutional layer. For activation function, Let m be the pixel value of the c-th channel in the output tensor of the l-th 3D convolutional layer, located at the i-th row and j-th column of the t-th frame, where m ≤ C. l-1 -1, δ t ≤K t -1, δ h ≤K h -1, δ w ≤K w -1, and i, j, t, s t s h s w m, C l-1 K t K h K h δ t δ h δ w c and l are all integers.

[0072] According to an embodiment of the present invention, the pixel value of each position of the first 3D feature map of each channel of the output of each 3D convolutional layer can be determined by the above formula (1), thereby obtaining the first 3D feature map.

[0073] According to one embodiment of the present invention, in order to enable the final output spatiotemporal feature map to integrate global and local feature information, the first 3D feature maps output by different 3D convolutional layers can be fused using a 3D feature pyramid structure. The first 3D feature maps output by shallow 3D convolutional layers have smaller receptive fields for pixels and contain more local feature information, while the first 3D feature maps output by deep 3D convolutional layers have larger receptive fields for pixels and contain more global feature information. Therefore, the first 3D feature maps output by multiple 3D convolutional layers can be fused to obtain a spatiotemporal feature map containing both global and local feature information.

[0074] According to one embodiment of the present invention, the first 3D feature map output from each 3D convolutional layer is subjected to layer-by-layer feature fusion to obtain a spatiotemporal feature map with multiple feature channels, including: processing the first 3D feature map of each layer through a 3D convolutional kernel with a kernel size of 1×1×1 to obtain a second preset number of second 3D feature maps for each layer; and recursively processing the second preset number of second 3D feature maps for each layer according to the size of the second 3D feature map of each layer and the number of layers to obtain a spatiotemporal feature map with multiple feature channels, wherein the number of feature channels of the spatiotemporal feature map is the second preset number.

[0075] According to one embodiment of the present invention, the number of output channels of different 3D convolutional layers may be different. In order to fuse the first 3D feature maps output by different 3D convolutional layers, the first 3D feature maps output by each 3D convolutional layer can be processed by a 3D convolutional kernel with a kernel size of 1×1×1, and the number of channels of the first 3D feature maps of each layer is unified (i.e., unified to a second preset number) to obtain the processed second 3D feature map. The second 3D feature map can be obtained by the following formula (2):

[0076] L (l) =Conv3D(F (l) W l lat (2)

[0077] Among them, F (l) F is the first 3D feature map of the l-th 3D convolutional layer. (l) The dimension is T l ×H l ×W l ×C l T lT is the number of frames in the first 3D feature map output by the l-th 3D convolutional layer. l =T, that is, the first preset quantity, H l W is the height of the first 3D feature map in each frame output by the l-th 3D convolutional layer. l C is the width of the first 3D feature map in each frame output by the l-th 3D convolutional layer. l The number of output channels in the l-th 3D convolutional layer, W l lat W represents the parameters of a 3D convolution kernel with a kernel size of 1×1×1. l lat The dimension is 1×1×1×C l ×C′, where C′ is the second preset quantity. L (l) This represents the second 3D feature map of the l-th layer. Conv3D indicates 3D convolution processing.

[0078] According to an embodiment of the present invention, the number of channels in each level can be unified through the above formula (2), thereby enabling the fusion of second 3D feature maps of different levels with the same channel to obtain spatiotemporal feature maps of multiple (second preset number) feature channels. During fusion, a layer-by-layer recursive fusion method can be adopted. Based on the size and number of levels of the second 3D feature map of each level, the second preset number of second 3D feature maps of each level are recursively processed to obtain spatiotemporal feature maps of multiple feature channels, including: when l = L, setting the second 3D feature map of the Lth level as the third 3D feature map of the Lth level, where L is the number of 3D convolutional levels; upsampling the third 3D feature map of the Lth level to obtain a fourth 3D feature map with the same size as the second 3D feature map of the L-1th level; and combining the fourth 3D feature map of the Lth level with the second 3D feature map of the L-1th level. The 3D feature maps are fused to obtain the third 3D feature map of the (L-1)th level; when l≠L, the third 3D feature map of the l-th level is upsampled to obtain a fourth 3D feature map with the same size as the second 3D feature map of the (L-1)th level; the fourth 3D feature map of the l-th level and the second 3D feature map of the (L-1)th level with the same feature channels are fused to obtain the third 3D feature map of the (L-1)th level; when the size of the third 3D feature map reaches the preset size, the recursion is stopped, and the third 3D feature map of multiple feature channels is determined as the spatiotemporal feature map of multiple feature channels.

[0079] According to one embodiment of the present invention, the sizes of the first 3D feature maps output by different 3D convolutional layers are different, for example, T l+1 =T l =T,H l+1W is the height of the first 3D feature map in each frame output by the (l+1)th 3D convolutional layer. l+1 H is the width of the first 3D feature map in each frame output by the (l+1)th 3D convolutional layer. l+1 The number of frames is the first 3D feature map output by the (l+1)th 3D convolutional layer. That is, except for the number of frames, the height and width of the first 3D feature map at different layers change. Furthermore, the above method of using a 1×1×1 3D convolutional kernel does not change the size of the feature map. Therefore, the size of the second 3D feature map is the same as the size of the first 3D feature map at the same layer. That is, the sizes of the second 3D feature maps at different layers are different from each other, and the size of the second 3D feature map at the last (Lth) layer is the smallest.

[0080] According to one embodiment of the present invention, starting from the second 3D feature map of the Lth level, the second 3D feature map of the Lth level is directly set as the third 3D feature map of the Lth level, and the third 3D feature map of the Lth level is upsampled to obtain a fourth 3D feature map with the same size as the second 3D feature map of the (L-1)th level. The fourth 3D feature map of the Lth level and the second 3D feature map of the (L-1)th level with the same feature channels are fused (for example, by adding them pixel by pixel, or by directly stitching them together to obtain more feature maps) to obtain the third 3D feature map of the (L-1)th level.

[0081] According to one embodiment of the present invention, if l≠L, the third 3D feature map of the l-th level is upsampled to obtain a fourth 3D feature map with the same size as the second 3D feature map of the (l-1)-th level. The fourth 3D feature map of the l-th level and the second 3D feature map of the (l-1)-th level with the same feature channels are then fused (e.g., by adding pixel by pixel, or by directly stitching them together to obtain more frame feature maps) to obtain the third 3D feature map of the (l-1)-th level. The above fusion process can be represented by the following formula (3):

[0082] P (l-1) =Up(P (l) )+L (l-1) (3)

[0083] Among them, L (l-1) For the second 3D feature map of the (l-1)th level, P (l-1) For the third 3D feature map of the (l-1)th level, P (l) Let Up represent the second 3D feature map of the l-th level, where Up(P) represents the upsampling process. (l) ) represents the fourth 3D feature map of the l-th level. When l = L, P (l) =L (l) .

[0084] According to an embodiment of the present invention, the above process can be executed iteratively. When the size of the third 3D feature map reaches a preset size (e.g., the size of the first 3D feature map at a certain level), the recursion can be stopped, and the third 3D feature map of the preset size can be used as a spatiotemporal feature map. The number of channels of the spatiotemporal feature map is a second preset number.

[0085] In this way, feature maps of different scales are obtained through multiple 3D convolutional layers, and feature maps of different scales are fused through a feature pyramid. The resulting spatiotemporal feature map contains rich local and global feature information, which is beneficial for understanding video content and improving the accuracy of anomaly type recognition results.

[0086] Example 2:

[0087] According to an embodiment of the present invention, in step S3, when performing video content understanding processing and determining the anomaly type recognition result, a Transformer-based model can be used for processing. Before using the model, the spatiotemporal feature map can be partitioned, and the information obtained from the partitions can be used as the input information of the Transformer-based model, i.e., multiple tokens. The attention mechanism of the Transformer model is then used to fuse the information of each partition to further understand the local and global feature information and improve the accuracy of the anomaly type recognition result.

[0088] According to one embodiment of the present invention, obtaining a partitioned spatiotemporal feature vector corresponding to multiple partitions of the spatiotemporal feature map includes: dividing the spatiotemporal feature map along the temporal direction, width direction, and height direction to obtain multiple partitioned feature maps; flattening the partitioned feature maps into partitioned feature vectors; determining the spatiotemporal position code corresponding to the partitioned feature map based on the sequence number of the partitioned feature map in the temporal feature map and its sequence number in the width direction and height direction; and fusing the spatiotemporal position code with the partitioned feature vector to obtain a partitioned spatiotemporal feature vector.

[0089] According to one embodiment of the present invention, the size of each partition feature map can be set, for example, the size of each partition feature map is T. p ×P h ×P w In the example, T p =2, P h =P w =16, meaning that the temporal direction of each spatiotemporal feature map can be divided into n t =T T / T p The height direction can be divided into n parts. h =H T / Ph The width direction can be divided into n parts. w =W T / P w T T H represents the number of frames in the spatiotemporal feature map. T W represents the height of the spatiotemporal feature map for each frame. T This represents the width of the spatiotemporal feature map for each frame. The set of partition indices is I = {(x,y,z)|0≤x}. <n t ,0≤y <n h ,0≤z <n w}, where x, y, and z are all integers. For the (x, y, z)th partition, xT represents the temporal direction in the spatiotemporal feature map. p up to (x+1)T p The range of -1, yP in the height direction h to (y+1)P h The range of -1, zP in the width direction w To (z+1)P w The area enclosed by the range of -1.

[0090] According to one embodiment of the present invention, the partition feature map of each partition can be flattened into a partition feature vector. That is, the pixel values ​​of each pixel in the partition feature map are sequentially used as data in the partition feature vector to obtain the partition feature vector corresponding to each partition. The length of the partition feature vector is T. p ×P h ×P w Furthermore, the above processing can be performed on the spatiotemporal feature maps of multiple feature channels, and multiple partition feature vectors can be obtained for each feature channel.

[0091] According to one embodiment of the present invention, a spatiotemporal position code corresponding to the partition feature map can be determined based on the temporal direction index (e.g., the xth index) and the indexes in the height and width directions (e.g., the (y, z)th index) of the partition feature map in the spatiotemporal feature map. For example, the temporal direction index is encoded to obtain a temporal position code; for example, x is processed through a fully connected layer to obtain a vector, which serves as the temporal position code. The spatial index is encoded to obtain a spatial position code; for example, (y, z) is processed through a fully connected layer to obtain a vector, which serves as the spatial position code. The temporal position code and the spatial position code are summed or concatenated to obtain the spatiotemporal position code. Further, the spatiotemporal position code can be concatenated or summed with the partition feature vector at the corresponding position to obtain the partition spatiotemporal feature vector.

[0092] In this way, the spatiotemporal feature map can be partitioned to obtain partition feature vectors suitable for the Transformer model. The partition feature vectors can be encoded according to their temporal and spatial positions, which makes it easier for the model to determine the position of each partition, thereby more accurately understanding the content of the video and improving the accuracy of the abnormal type recognition results.

[0093] Example 3:

[0094] According to an embodiment of the present invention, in step S4, the encoding model is the Transformer-based model described above. Multiple partitioned spatiotemporal feature vectors from the above multiple channels can be used as input information to the encoding model to obtain comprehensive spatiotemporal feature information. That is, the multiple partitioned spatiotemporal feature vectors from multiple channels are fused to obtain comprehensive spatiotemporal feature information that integrates multiple scales and local and global feature information.

[0095] According to one embodiment of the present invention, the spatiotemporal feature vectors corresponding to multiple partitions are input into an encoding model for processing to obtain comprehensive spatiotemporal feature information. This includes: processing the input feature vectors using a multi-head attention mechanism at each level of the encoding model to obtain the output feature information of each attention head, wherein in the first level of the encoding model, the input feature vector is the spatiotemporal feature vector of the partition; concatenating the output feature information of each attention head to obtain concatenated feature information; processing the concatenated feature information using a weight matrix to obtain an output feature vector, and using the output feature vector as the input feature vector for the next level; and combining the output feature vectors after processing through multiple levels of the encoding model to obtain comprehensive spatiotemporal feature information.

[0096] According to one embodiment of the present invention, each layer of the encoding model is a Transformer-based model. In each layer, a multi-head attention mechanism is used to process the input feature vectors to obtain the spatial and temporal dependencies between different input feature vectors (tokens), thereby achieving the fusion of spatiotemporal information.

[0097] According to one embodiment of the present invention, a query weight matrix W can be generated for each attention head. Q Key-value weight matrix W K , and numerical weight matrix W VThe input feature vectors of this level are combined into a matrix. For example, in the first level, multiple partitioned spatiotemporal feature vectors are used as input feature vectors and combined into a matrix. If the number of levels is not 1, the output feature vector of the previous level is used as the input feature vector and combined into a matrix. This matrix is ​​used as the input matrix. The input matrix is ​​used to perform matrix multiplication with the query weight matrix, key value weight matrix and numerical weight matrix respectively to obtain the query matrix, key value matrix and numerical matrix corresponding to the attention head. That is, the query matrix Q corresponding to the a-th attention head is obtained according to the following formulas (4), (5) and (6). a Key matrix K a sum numerical matrix V a ,

[0098]

[0099]

[0100] Among them, H (l-1) Let be the input matrix for the l-th level of the encoding model. Let be the query weight matrix corresponding to the a-th attention head. Let be the key-weight matrix corresponding to the a-th attention head. Let be the numerical weight matrix corresponding to the a-th attention head.

[0101] According to an embodiment of the present invention, the output feature information head of the a-th attention head can be obtained according to the following formula (7). a :

[0102]

[0103] in, For K a The transpose of d k For K a The dimension of each vector in the vector is denoted by Softmax, which is the activation function.

[0104] According to an embodiment of the present invention, the output feature information corresponding to each attention head can be obtained through the above formulas (4)-(7), and then the output feature information can be concatenated to obtain concatenated feature information. Furthermore, the concatenated feature information can be processed by a weight matrix to obtain an output feature vector, as shown in the following formula (8):

[0105] Multihead_Attention(H (l-1) = Concat(head1,…,head) h )w O (8)

[0106] Where head1 is the output feature information of the first attention head, head h For the output feature information of the h-th attention head, w O For the weight matrix, Multihead_Attention(H) (l-1) The result of the multi-head attention mechanism processing the input feature vector is shown in the diagram. `Concat` represents concatenating the output feature information from multiple attention heads. Further, the result of the multi-head attention mechanism is normalized and processed by multiple network layers (e.g., linear operation layers and activation layers) to obtain an output matrix, which includes multiple output feature vectors and can be used as the input feature vector for the next layer.

[0107] According to one embodiment of the present invention, after multiple levels of processing, the resulting output matrix (i.e., a matrix composed of multiple output feature vectors) can be used as comprehensive spatiotemporal feature information.

[0108] In this way, the spatiotemporal feature vectors of multiple channels can be fused through a multi-head attention mechanism to obtain comprehensive spatiotemporal feature information that integrates multiple scales and local and global feature information. This allows us to obtain the spatial and temporal dependencies between different input feature vectors, thereby more accurately understanding video content and improving the accuracy of anomalous type recognition results.

[0109] Example 4:

[0110] According to one embodiment of the present invention, in step S5, the spatiotemporal feature information can be processed by a classification and discrimination model to obtain the anomaly type identification result. The classification and discrimination model can be a multilayer perceptron model (e.g., a model composed of multiple fully connected layers and activation layers), which can perform fusion, dimensionality reduction, and other processing on the comprehensive spatiotemporal feature information, and finally output the probability that a first preset number of consecutive video frames to be processed belong to various anomaly types. Furthermore, regularization, random deactivation, and other methods can be combined to improve the robustness and generalization ability of the classification.

[0111] According to one embodiment of the present invention, inputting the comprehensive spatiotemporal feature information into a classification and discrimination model for processing to obtain anomaly type identification results includes: inputting the comprehensive spatiotemporal feature information into a multi-layer perception layer of the classification and discrimination model for processing to obtain anomaly type feature vectors; and inputting the anomaly type feature vectors into the activation layer of the classification and discrimination model for processing to obtain anomaly type identification results.

[0112] According to an embodiment of the present invention, the heteromorphic type feature vector V can be obtained according to the following formula (9):

[0113] V=W2σ(W1H C+b1)+b2(9)

[0114] Where W1 and W2 are weight matrices, H C To integrate spatiotemporal feature information, b1 and b2 are bias vectors, and σ is the activation function.

[0115] According to one embodiment of the present invention, the first fully connected layer can process the integrated spatiotemporal feature information through W1 and b1. After processing by the activation function, the second fully connected layer can further process the result of the activation function through W2 and b2 to obtain the heterogeneous feature vector V.

[0116] According to one embodiment of the present invention, inputting the feature vector of the abnormal type into the activation layer of the classification and discrimination model for processing to obtain the abnormal type identification result includes: inputting the feature vector of the abnormal type into the activation layer of the classification and discrimination model for processing to output probability data of multiple abnormal types; and taking the abnormal type corresponding to the maximum value of the probability data as the abnormal type identification result.

[0117] According to an embodiment of the present invention, the probability data of the f-th anomaly type can be obtained according to the following formula (10):

[0118]

[0119] Among them, v f Let f be the f-th data in V, and v g Let be the g-th data in V, M be the number of heteromorphic types, and τ be the temperature coefficient greater than 0.

[0120] According to an embodiment of the present invention, after obtaining the probability data of each anomaly type based on formula (10), the anomaly type corresponding to the maximum value of the probability data can be used as the anomaly type identification result.

[0121] In this way, the comprehensive spatiotemporal feature information can be processed by the classification model to fully utilize the feature information at multiple scales, local and global feature information, and spatial and temporal dependencies contained in the comprehensive spatiotemporal feature information, so as to obtain the anomaly type identification results, improve the accuracy of the anomaly type identification results, and enhance the robustness and generalization ability of the classification.

[0122] Example 5:

[0123] Figure 2 An exemplary schematic diagram illustrates the application of an artificial intelligence identification method for detecting anomalies in audio and video broadcast content according to an embodiment of the present invention.

[0124] According to one embodiment of the present invention, multiple video frames to be processed can be used to extract features through multiple 3D convolutional layers, and the obtained feature maps can be fused through a 3D feature pyramid structure to obtain a spatiotemporal feature map. The spatiotemporal feature map can be partitioned along the time, height, and width directions to obtain partitioned feature maps, and after flattening, a partitioned feature vector for each partitioned feature map can be obtained. Furthermore, based on the position of the partitioned feature map in the spatiotemporal feature map, a spatiotemporal position code can be obtained. After fusing the spatiotemporal position code with the partitioned feature vector, the input feature vector of the encoding model, i.e., multiple tokens, can be obtained.

[0125] According to one embodiment of the present invention, the encoding model is a transformer-based model, which may include multiple layers. Each layer can perform layer normalization on the input feature vector, and then obtain the output feature information corresponding to each attention head through a multi-head attention mechanism. After concatenating, weighting, layer normalization, and multi-layer perception of the output feature information corresponding to each attention head, an output feature vector is obtained, which is used as the input feature vector of the next layer. After iterating C times, comprehensive spatiotemporal feature information is obtained. By processing the comprehensive spatiotemporal feature information through a classification and discrimination model, probability data of various anomalies can be obtained, and the anomaly type corresponding to the maximum probability data is taken as the anomaly type identification result.

[0126] The AI-based method for detecting anomalies in audio and video broadcast content according to embodiments of the present invention can effectively capture local and global information in the video through feature extraction processing at multiple levels and in multiple partitions. This facilitates accurate understanding of the video content, enabling the identification of anomaly types across multiple consecutive frames of the video. Furthermore, it is applicable to a wider range of scenarios and anomaly types, improving both accuracy and applicability. Further, feature maps of different scales can be obtained through multiple 3D convolutional layers, and these feature maps can be fused using a feature pyramid approach. This results in a spatiotemporal feature map containing rich local and global feature information, which is beneficial for understanding the video content and improving the accuracy of anomaly type identification. The spatiotemporal feature map can also be partitioned to obtain partition feature vectors suitable for the Transformer model. Encoding based on the temporal and spatial positions of these partition feature vectors facilitates model determination of the location of each partition, leading to a more accurate understanding of the video content and further improving the accuracy of anomaly type identification. Furthermore, a multi-head attention mechanism can be used to fuse spatiotemporal feature vectors from multiple channels and partitions, resulting in comprehensive spatiotemporal feature information that integrates multiple scales and local and global feature information. This allows for the acquisition of spatial and temporal dependencies between different input feature vectors, leading to a more accurate understanding of video content and improved accuracy in anomaly identification. Further, a classification model can be used to process the comprehensive spatiotemporal feature information to fully utilize the multi-scale, local, and global feature information, as well as spatial and temporal dependencies contained within it, thereby obtaining anomaly identification results. This improves the accuracy of anomaly identification and enhances the robustness and generalization ability of the classification.

[0127] Figure 3 An exemplary block diagram of a system for artificial intelligence identification of anomalies in audio and video broadcast content according to an embodiment of the present invention is shown, the system comprising:

[0128] The decoding module is used to decode the video to be processed and obtain multiple video frames to be processed.

[0129] The feature extraction module is used to perform feature extraction processing on a first preset number of consecutive video frames to be processed, and obtain spatiotemporal feature maps of multiple feature channels.

[0130] The partitioning module is used to obtain partitioned spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map based on the spatiotemporal feature map.

[0131] The encoding module is used to input the spatiotemporal feature vectors of multiple partitions into the encoding model for processing to obtain comprehensive spatiotemporal feature information;

[0132] The identification module is used to input the comprehensive spatiotemporal feature information into the classification and discrimination model for processing, and obtain the anomaly type identification result.

[0133] According to one embodiment of the present invention, a computer-readable storage medium is provided having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method for artificial intelligence recognition of anomalies in audio and video broadcast content.

[0134] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0135] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting anomalies in audio and video broadcast content using artificial intelligence, characterized in that, include: The video to be processed is decoded to obtain multiple video frames to be processed; Feature extraction is performed on a first preset number of consecutive video frames to be processed to obtain spatiotemporal feature maps of multiple feature channels; Based on the spatiotemporal feature map, obtain the partition spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map; The spatiotemporal feature vectors of multiple partitions are input into the encoding model for processing to obtain comprehensive spatiotemporal feature information. The comprehensive spatiotemporal feature information is input into a classification and discrimination model for processing to obtain the anomaly type identification result; Feature extraction is performed on a first preset number of consecutive video frames to be processed to obtain spatiotemporal feature maps with multiple feature channels, including: By using multiple 3D convolutional layers, a layer-by-layer feature extraction process is performed on a first preset number of consecutive video frames to be processed to obtain the first 3D feature map output by each 3D convolutional layer. The first 3D feature map output from each 3D convolutional layer is fused layer by layer to obtain spatiotemporal feature maps with multiple feature channels. The first 3D feature map output from each 3D convolutional layer is fused layer by layer to obtain spatiotemporal feature maps with multiple feature channels, including: The first 3D feature map of each level is processed by a 3D convolution kernel with a kernel size of 1×1×1 to obtain a second preset number of second 3D feature maps of each level. Based on the size and number of layers of the second 3D feature map of each layer, the second preset number of second 3D feature maps of each layer are recursively processed to obtain spatiotemporal feature maps with multiple feature channels, wherein the number of feature channels of the spatiotemporal feature map is the second preset number. Based on the size and number of layers of the second 3D feature map at each level, a predetermined number of second 3D feature maps at each level are recursively processed to obtain spatiotemporal feature maps with multiple feature channels, including: exist l = L At that time, the first L The second 3D feature map of the first level is set to the first L The third 3D feature map of each level, among which, L The number of 3D convolutional layers; The first L Upsampling is performed on the third 3D feature map of the first level to obtain the same result as the first level. L -A fourth 3D feature map of the same size as the second 3D feature map of the -1 level; The first with the same feature channel L The fourth 3D feature map of the first level and the first L -1 level of the second 3D feature map is fused to obtain the first... L -1 level of third 3D feature map; exist l ≠ L At that time, the first l Upsampling is performed on the third 3D feature map of the first level to obtain the same result as the first level. l -A fourth 3D feature map of the same size as the second 3D feature map of the -1 level; The first with the same feature channel l The fourth 3D feature map of the first level and the first l -1 level of the second 3D feature map is fused to obtain the first... l -1 level of third 3D feature map; When the size of the third 3D feature map reaches the preset size, the recursion stops, and the third 3D feature map of multiple feature channels is determined as the spatiotemporal feature map of multiple feature channels.

2. The method according to claim 1, characterized in that, Based on the spatiotemporal feature map, partition spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map are obtained, including: The spatiotemporal feature map is divided along the temporal direction, width direction, and height direction to obtain multiple partition feature maps; Flatten the partition feature map into partition feature vectors; Based on the temporal direction index of the partition feature map in the spatiotemporal feature map, as well as the indexes in the width and height directions, determine the spatiotemporal position code corresponding to the partition feature map; The spatiotemporal location code is fused with the partition feature vector to obtain the partition spatiotemporal feature vector.

3. The method according to claim 1, characterized in that, The spatiotemporal feature vectors corresponding to multiple partitions are input into the encoding model for processing to obtain comprehensive spatiotemporal feature information, including: In each level of the encoding model, a multi-head attention mechanism is used to process the input feature vector to obtain the output feature information of each attention head. In the first level of the encoding model, the input feature vector is the partitioned spatiotemporal feature vector. The output feature information of each attention head is concatenated to obtain concatenated feature information; The concatenated feature information is processed by a weight matrix to obtain an output feature vector, which is then used as the input feature vector for the next level. After processing through multiple levels of the encoding model, the output feature vectors are combined to obtain comprehensive spatiotemporal feature information.

4. The method according to claim 1, characterized in that, The comprehensive spatiotemporal feature information is input into a classification and discrimination model for processing to obtain the anomaly type identification result, including: The comprehensive spatiotemporal feature information is input into the multi-layer perception layer of the classification and discrimination model for processing to obtain the anomaly type feature vector; The feature vectors of the abnormal type are input into the activation layer of the classification and discrimination model for processing to obtain the abnormal type recognition result.

5. The method according to claim 4, characterized in that, The anomaly type feature vector is input into the activation layer of the classification and discrimination model for processing to obtain the anomaly type recognition result, including: The feature vectors of the heterogeneous types are input into the activation layer of the classification and discrimination model for processing, and the probability data of various heterogeneous types are output. The anomaly type corresponding to the maximum probability data is taken as the anomaly type identification result.

6. A system for detecting anomalies in broadcast audio and video content using artificial intelligence, for performing the method as described in any one of claims 1-5, characterized in that, include: The decoding module is used to decode the video to be processed and obtain multiple video frames to be processed. The feature extraction module is used to perform feature extraction processing on a first preset number of consecutive video frames to be processed, and obtain spatiotemporal feature maps of multiple feature channels. The partitioning module is used to obtain partitioned spatiotemporal feature vectors corresponding to multiple partitions of the spatiotemporal feature map based on the spatiotemporal feature map. The encoding module is used to input the spatiotemporal feature vectors of multiple partitions into the encoding model for processing to obtain comprehensive spatiotemporal feature information; The identification module is used to input the comprehensive spatiotemporal feature information into the classification and discrimination model for processing, and obtain the anomaly type identification result.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method as described in any one of claims 1-5.

Images