Video understanding method, device, computer equipment and computer readable storage medium

Through the spatial pyramid model and the timing pyramid model, video frames are processed and multi-scale features are extracted, which solves the problem of inefficiency of existing video comprehension methods and achieves efficient and accurate video comprehension.

CN118658105BActive Publication Date: 2024-11-26华润数字科技有限公司 +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411125358.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

Existing video comprehension methods are inefficient, requiring a large amount of preprocessing and sample data, and deep learning algorithms and human experts' evaluation resources are consumed, resulting in inefficient understanding of videos.

Method used

The spatial pyramid model is used to determine the apparent characteristics of the video frame image on multiple spatial scales, and these characteristics are grouped through the timing pyramid model to obtain the behavioral timing relationships on different time scales, and finally fusion is carried out to improve the video understanding efficiency and accuracy.

Benefits of technology

The behavioral characteristics of videos of multiple scales can be quickly extracted without pre-processing, which improves the efficiency and accuracy of video comprehension and reduces the consumption of computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118658105B_ABST
    Figure CN118658105B_ABST
Patent Text Reader

Abstract

The present application provides a video understanding method, device, computer equipment and computer-readable storage medium, which belongs to the field of image processing technology. The video understanding method includes: determining the apparent features of each frame of the video to be understood at multiple spatial scales through a spatial pyramid model; grouping and processing the apparent features of at least part of the images in the video to be understood at the multiple spatial scales through a temporal pyramid model to obtain multiple behavioral temporal relationships of the video to be understood; the multiple behavioral temporal relationships have different time scales; and fusing the multiple behavioral temporal relationships to obtain the understanding result of the video to be understood. By adopting the present application, the video content can be understood more efficiently and accurately.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more particularly to a video understanding method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] Nowadays, there are many ways to understand videos. For example, image understanding can be performed by quickly extracting visual features from video images, or machine learning algorithms can be used to automatically learn features and patterns in videos to classify and identify videos.

[0003] However, the method of extracting visual features for image understanding requires a lot of preprocessing of the video image in the early stage, and a large number of visual features need to be selected in the actual process of extracting visual features, which leads to low efficiency in understanding the video. In addition, when using machine learning algorithms for video image understanding, a large amount of sample data and time are required for model training, which also leads to low efficiency in understanding the video.

[0004] In summary, existing methods for understanding videos are generally inefficient. Summary of the invention

[0005] The embodiments of the present application provide a video understanding method, apparatus, computer device and computer-readable storage medium, aiming to improve the efficiency of video understanding.

[0006] To achieve the above object, a first aspect of an embodiment of the present application provides a video understanding method, the method comprising:

[0007] Determine the apparent features of each frame of the video to be understood at multiple spatial scales through a spatial pyramid model;

[0008] The appearance features of at least part of the images in the video to be understood are grouped at the multiple spatial scales by using a temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood, wherein the multiple behavior temporal relationships have different time scales;

[0009] The multiple behavior temporal relationships are integrated to obtain an understanding result of the video to be understood.

[0010] In some embodiments, determining the apparent features of each frame of the video to be understood at multiple spatial scales by using a spatial pyramid model includes:

[0011] Performing multi-layer convolution processing on each frame image of the video to be understood through a spatial pyramid model to obtain multi-layer convolution features of each frame image;

[0012] Determine C target layer convolution features from the multi-layer convolution features in the order of the convolution layers in the spatial pyramid model from high to low; C is a positive integer greater than 1, and C is less than the number of convolution layers in the spatial pyramid model;

[0013] The C target layer convolutional features are fused to obtain the apparent features of each frame of the image at multiple spatial scales.

[0014] In some embodiments, the C target layer convolutional features include: top layer convolutional features and non-top layer convolutional features;

[0015] The fusing of the C target layer convolution features to obtain the apparent features of each frame of image at multiple spatial scales includes:

[0016] Pooling the top-level convolutional features and the non-top-level convolutional features to obtain pooled top-level convolutional features and pooled non-top-level convolutional features;

[0017] Adjusting the pooled non-top convolution features through a one-dimensional convolution layer to obtain adjusted non-top convolution features;

[0018] The pooled top-level convolutional features and the adjusted non-top-level convolutional features are fused to obtain the apparent features of each frame of the image at multiple spatial scales.

[0019] In some embodiments, the temporal pyramid model includes: a frame pooling layer and a relational reasoning layer;

[0020] The process of grouping the appearance features by the temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood includes:

[0021] The method of grouping the apparent features of at least part of the images in the video to be understood at the multiple spatial scales by using the temporal pyramid model to obtain multiple temporal relationships of behaviors in the video to be understood includes:

[0022] Dividing at least part of the images in the video to be understood into multiple groups of video segments;

[0023] The frame pooling layer is used to pool the apparent features of the images contained in the multiple groups of video clips respectively, so as to obtain multiple video clip features; the apparent features contained in the multiple groups of video clips are the apparent features of the image frames contained in the multiple groups of video clips respectively;

[0024] The relational reasoning layer performs relational reasoning on the multiple video clip features to obtain multiple behavior temporal relationships.

[0025] In some embodiments, the number of the frame pooling layers is multiple, and the time intervals of the multiple frame pooling layers are different; the time ranges of the multiple groups of video clips are the same;

[0026] The step of pooling the apparent features of the images respectively contained in the plurality of groups of video clips through the frame pooling layer comprises:

[0027] Calling a target pooling layer in the pooling layer based on the time range; a time interval of the target pooling layer corresponds to the time range;

[0028] The target pooling layer is used to pool the apparent features of the images contained in each of the multiple groups of video clips.

[0029] In some embodiments, performing relational reasoning on the multiple video clip features through the relational reasoning layer to obtain multiple behavior temporal relationships includes:

[0030] Connecting video segment features with the same time interval in series among the multiple video segment features to obtain multiple series features;

[0031] The relational reasoning layer performs relational reasoning on the multiple series features respectively to obtain multiple behavior temporal relationships.

[0032] In some embodiments, the number of the relational reasoning layers is multiple, and the time intervals corresponding to the multiple relational reasoning layers are different; the time intervals of the multiple series features are different;

[0033] The performing relational reasoning on the plurality of series features respectively through the relational reasoning layer includes:

[0034] Performing relational reasoning on a target series feature among the multiple series features through a target series reasoning layer among the multiple series reasoning layers;

[0035] The time interval corresponding to the target relationship reasoning layer is the same as the time interval of the target series features.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video understanding device, the device comprising:

[0037] A spatial pyramid module is used to determine the apparent features of each frame of the video to be understood at multiple spatial scales through a spatial pyramid model;

[0038] A temporal pyramid module, used for grouping and processing the apparent features of at least part of the images in the video to be understood at the multiple spatial scales through a temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood; the multiple behavior temporal relationships each have a different time scale;

[0039] The fusion module is used to fuse the multiple behavior temporal relationships to obtain the understanding result of the video to be understood.

[0040] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides a computer device, the device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0041] Memory, used to store computer programs;

[0042] The processor is used to implement the steps of the video understanding method described in the first aspect when executing the program stored in the memory.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video understanding method described in the first aspect are implemented.

[0044] To achieve the above-mentioned purpose, the fifth aspect of the embodiments of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video understanding method described in the first aspect above.

[0045] The video understanding method, device, computer equipment, computer-readable storage medium and computer program product provided by the embodiments of the present application determine the apparent features of each frame image in the video to be understood at multiple spatial scales through a spatial pyramid model, and group the apparent features through a temporal pyramid model, thereby obtaining the behavioral temporal relationship of the video to be understood at multiple different time scales, and then fuse the multiple behavioral temporal relationships at different time scales to obtain the understanding result for the entire video. In this way, the present application does not need to pre-process the video in the early stage, but only needs to use the video to be understood directly as input, and can use the spatial feature pyramid model and the temporal pyramid model to process the video successively, and finally obtain the behavioral features of the video to be understood at multiple scales, thereby improving the efficiency of understanding the video content. In addition, the present application extracts the features of the video content at different spatiotemporal scales for analysis and understanding through the spatial feature pyramid model and the temporal pyramid model, which also improves the accuracy of the results of video understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A schematic diagram of the steps of the video understanding method provided in some embodiments of the present application;

[0048] Figure 2 A schematic diagram of the structural framework of a terminal device involved in the video understanding method provided in an embodiment of the present application;

[0049] Figure 3 A schematic flow chart of the detailed steps of determining the appearance features involved in the video understanding method provided in the embodiment of the present application;

[0050] Figure 4 A schematic flow chart of the refinement steps of the fused target layer convolutional features involved in the video understanding method provided in the embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a spatial pyramid module involved in the video understanding method provided in an embodiment of the present application;

[0052] Figure 6 A schematic flow chart of the refinement steps of processing appearance features involved in the video understanding method provided in the embodiment of the present application;

[0053] Figure 7 A schematic diagram of a scene of frame timing pooling involved in the video understanding method provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of a temporal pyramid module involved in the video understanding method provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of the structure of a video understanding device provided in an embodiment of the present application;

[0056] Fig.10 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] Before introducing the embodiments of the present application, a brief introduction to the relevant technical terms involved in the embodiments of the present application is first given here.

[0061] Video understanding.

[0062] Video understanding can also be called video recognition. Video understanding aims to automatically identify and analyze the content in the video through intelligent analysis technology. Video understanding involves many aspects of life. At present, video understanding has developed into a very broad academic research and industrial application direction, among which the most important ones are action recognition (Action Recognition), temporal action localization (Temporal Action Localization) and video vectorization (Vedio Embedding).

[0063] The goal of action recognition is to identify the actions that appear in the video (usually the actions of people in the video). Since the video can be regarded as a data structure composed of a set of image frames arranged in time order, which has an additional time dimension than the image, action recognition not only needs to analyze the content of each frame in the video, but also needs to mine clues from the timing information between the video frames. Temporal Action Localization, also known as Temporal Action Detection, is another important area of ​​video understanding. Temporal action localization not only predicts what actions are contained in the video, but also predicts the start and end time of the action. Compared with action recognition, temporal action localization is closer to the real scene. The goal of video vectorization is to obtain a low-dimensional, dense, floating-point feature vector representation from the video. This feature vector is a summary and generalization of the entire video content.

[0064] Scale Space Theory.

[0065] Scale Space Theory is an important concept in the field of computer vision and image processing. It provides a systematic method to analyze the characteristics of images at different scales. The core idea of ​​visual scale space is to regard the image as a continuous multi-scale space, where each scale represents the representation of the image at different resolutions. In the scale space, each point of the image is associated with a specific scale, which determines the characteristics of the local structure of the image at this point. By analyzing images at multiple scales, multi-level information from details to the global can be captured. In technical implementation, the scale space is usually constructed by Gaussian blurring of the image.

[0066] Scale space theory has a wide range of applications in many fields, such as image segmentation, feature extraction, and object recognition. For example, in image segmentation, scale space can help identify edges and textures of different scales; in feature extraction, algorithms such as Scale-Invariant Feature Transform (SIFT) use scale space to extract features that are robust to scale and rotation changes. In addition, scale space theory also provides a model for understanding the human visual system. In general, visual scale space theory provides a powerful tool for image analysis. It captures the features of images at multiple scales and enhances the algorithm's ability to understand the content of the image, thus playing an important role in many application fields of computer vision.

[0067] The basic idea of ​​scale space theory is to introduce scale parameters into the visual information processing model, obtain visual processing information at different scales by continuously changing the scale parameters, and then integrate this information to accurately represent the essential characteristics of the image. The scale space method incorporates the traditional single-scale visual information processing technology into a dynamic analysis framework with constantly changing scales, so it is easier to obtain the essential characteristics of the image.

[0068] Next, the overall concept of the video understanding method provided in the embodiment of the present application is further explained.

[0069] Nowadays, there are many ways to understand videos. For example, you can quickly extract visual features from video images to understand images, or you can use machine learning algorithms to automatically learn features and patterns in videos to classify and identify videos. However, the method of extracting visual features to understand images requires a lot of preprocessing of video images in the early stages, and a large number of visual features also need to be selected in the actual process of extracting visual features, which leads to relatively low efficiency in understanding videos. In addition, when using machine learning algorithms to understand videos, a large amount of sample data and time are required for model training, which also leads to low efficiency in understanding videos.

[0070] In addition, there are also ways to understand videos using deep learning algorithms or with the help of human expert evaluation. Among them, using deep learning algorithms for video understanding is to use deep neural networks to learn the features and patterns in the video, and to classify and identify them. Even if the deep learning algorithm can achieve automatic learning for video understanding, it still requires a large amount of training data and occupies a large amount of computing resources, resulting in low efficiency in understanding the video. With the help of human expert evaluation, human experts are asked to evaluate and judge the content in the video to determine whether the video image contains certain specific content. The advantage of this method is that it can integrate human wisdom and experience, so that users can accept the results of understanding the video more easily and can be recognized by users more easily. However, using human experts to understand the video will inevitably consume a large amount of human expert resources and time resources, which also leads to low efficiency in understanding the video.

[0071] In summary, although the existing methods for understanding videos have their own advantages, they all generally have the problem of low efficiency in understanding videos.

[0072] In view of this, an embodiment of the present application provides a video understanding method. First, the apparent features of each frame image in the video to be understood at multiple spatial scales are determined by a spatial pyramid model, and then the apparent features are grouped and processed by a temporal pyramid model, so as to obtain the behavioral temporal relationship at multiple different time scales of the video to be understood, and finally the multiple behavioral temporal relationships at different time scales are fused to obtain the understanding result for the entire video. In this way, the video understanding method provided by the embodiment of the present application does not need to pre-process the video in the early stage, but only needs to use the video to be understood directly as input, and can use the spatial feature pyramid model and the temporal pyramid model to process the video successively, and finally obtain the behavioral features of multiple scales of the video to be understood. Thereby, the efficiency of understanding the video can be improved, and the accuracy of the result of understanding the video can also be improved.

[0073] It should be noted that the video understanding method provided in the embodiment of the present application can be applied to the terminal device. It should be understood that the video understanding method provided in the embodiment of the present application can also be applied to the server side. Alternatively, the video understanding method provided in the embodiment of the present application can also be software running on the terminal device or on the server side. In some embodiments, the terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and it can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video understanding method, etc., but is not limited to the above forms.

[0074] Alternatively, the present application may also be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types.

[0075] For ease of understanding and explanation, the following text takes the video understanding method provided by the embodiment of the present application as an example to explain the present application in detail. The implementation of the video understanding method provided by the embodiment of the present application in any of the above-mentioned forms of the subject application can refer to the process of the terminal device applying the video understanding method described below.

[0076] Please refer to Figure 1 , Figure 1 The following is a flow chart of the steps of the video understanding method provided in some embodiments of the present application. Figure 1 The execution order of some method steps is shown in the figure, but based on different design requirements of actual applications, the video understanding method provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1 The order of the steps of the method shown does not constitute a limitation on the execution logic order of the video understanding method provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the steps of the method shown should be included in the protection scope of the video understanding method provided in the embodiments of the present application.

[0077] like Figure 1 As shown, in some embodiments, the terminal device applies the video understanding method provided by the embodiment of the present application, which may include steps S101 to S103.

[0078] Step S101, determining the apparent features of each frame of the video to be understood at multiple spatial scales through the spatial pyramid model.

[0079] It should be noted that the spatial pyramid model can be constructed using the inherent multi-level information of the convolutional neural network. The spatial pyramid model has a bottom-up structure. The top layer can represent the semantic content of high-level behavior, but loses important local information of the behavior. The low-level features have a high spatial resolution and preserve important behavioral details, but lack abstract semantic content. In this way, the spatial pyramid model uses horizontal connections to fuse feature representations at different levels, and can obtain pyramid features with rich semantics at multiple spatial scales. The pyramid features of multiple spatial scales can represent the apparent characteristics of the behavior.

[0080] The terminal device inputs the video to be understood into a pre-built spatial pyramid model, and then processes each frame of the video to be understood through the spatial pyramid model, thereby obtaining a pyramid feature with rich semantics at multiple spatial scales for each frame of the video to be understood, and uses the pyramid feature to represent the apparent characteristics of the content in each frame of the image at multiple spatial scales.

[0081] Step S102, grouping and processing the apparent features of at least part of the images in the video to be understood at the multiple spatial scales by using the temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood; the multiple behavior temporal relationships each have a different time scale;

[0082] It should be noted that the temporal pyramid model can be a temporal pyramid network or a time pyramid network pre-configured for the terminal device. In some embodiments, the temporal pyramid model can be composed of four parts, such as a backbone network (mainly used to extract features), a pooling network (mainly used to pool features), a relational reasoning network (mainly used to reason about different temporal relationships), and a fusion network (used to fuse temporal relationships on different time scales to obtain output results). In other embodiments, the temporal pyramid model can also be composed of only a pooling network and a relational reasoning network.

[0083] After the terminal device determines the apparent features of each frame of the video to be understood through the spatial pyramid model, it further groups and pools the apparent features of at least part of the images in the video to be understood through the temporal pyramid model, and infers the behavioral temporal relationships at different time scales for the pooled features, thereby obtaining multiple behavioral temporal relationships of the video to be understood at different time scales.

[0084] It should be noted that when the terminal device groups and pools the appearance features of at least part of the images in the video to be understood through the temporal pyramid model, the terminal device first groups at least part of the images in the video to be understood, and then pools the appearance features contained in each group of images. Alternatively, the terminal device may also group the video feature sequences composed of the appearance features of at least part of the images in the video to be understood, and then pool each group of feature sequences.

[0085] In addition, the terminal device can group and pool the appearance features of all images in the video to be understood through the temporal pyramid model, or the terminal device can group and pool the appearance features of a part of the images in the video to be understood through the temporal pyramid model. In order to ensure a comprehensive understanding of the video content, when the terminal device group and pool the appearance features of a part of the images in the video to be understood through the temporal pyramid model, it can extract half of all images in the video to be understood by using an alternate frame processing method, and then group and pool the appearance features of the half of the images.

[0086] Step S103: fuse the multiple behavior temporal relationships to obtain an understanding result of the video to be understood.

[0087] It should be noted that the understanding result may be a temporal structure in which the behavior in the video to be understood is represented by multiple time scales.

[0088] After the terminal device processes the apparent features of each frame of the video to be understood through the temporal pyramid model to obtain multiple behavioral temporal relationships of the video to be understood at different time scales, the terminal device further uses a horizontal connection to fuse the multiple behavioral temporal relationships at different time scales, thereby obtaining a temporal structure in which the behavior in the video to be understood is represented by multiple time scales.

[0089] For example, please refer to Figure 2 , Figure 2 A schematic diagram of the structural framework of a terminal device involved in the video understanding method provided in an embodiment of the present application.

[0090] like Figure 2As shown in the figure, when the terminal device is performing video understanding, it first inputs the video to be understood with multiple consecutive frames of images into the spatial pyramid module, and the spatial pyramid module calls the pre-configured spatial pyramid model to perform multiple levels of feature extraction processing on each frame of the image, so as to obtain pyramid features with rich semantics at multiple spatial scales for each frame of the video to be understood. In this way, the terminal device uses the pyramid features as the apparent features of the content of each frame of the video to be understood at multiple spatial scales.

[0091] After that, the spatial pyramid module outputs the video to be understood, from which the apparent features of each frame of the image have been extracted, and the video to be understood is automatically input into the temporal pyramid module. The temporal pyramid model calls the pre-configured temporal pyramid model to first pool the apparent features contained in each frame of the video to be understood, and then infers the behavior temporal relationship of different time scales based on the pooled features, thereby obtaining multiple behavior temporal relationships of the video to be understood at different time scales.

[0092] Finally, the temporal pyramid module outputs multiple temporal relationships of behaviors in the video to be understood, and the terminal device fuses the multiple temporal relationships of behaviors in the video to be understood in a horizontal connection manner, thereby obtaining a temporal structure of behaviors in the video to be understood represented by multiple time scales. In this way, the terminal device outputs the final temporal structure represented by multiple time scales as the understanding result of the video to be understood.

[0093] In the embodiment of the present application, the apparent features of each frame image of the video to be understood at multiple spatial scales are first determined by the spatial pyramid model, and then the apparent features are grouped and processed by the temporal pyramid model to obtain the behavioral temporal relationships of multiple different time scales of the video to be understood, and finally the behavioral temporal relationships of multiple different time scales are fused to obtain the understanding result of the video to be understood as a whole. In this way, since there is no need to pre-process the video in the early stage, and only the video to be understood needs to be directly used as input, the spatial feature pyramid model and the temporal pyramid model can be used to process the video successively, and finally the behavioral features of multiple scales of the video to be understood can be obtained, thereby improving the efficiency of understanding the video content. In addition, the features of the video content at different time and space scales are extracted by the spatial feature pyramid model and the temporal pyramid model for analysis and understanding, which also improves the accuracy of the results of video understanding.

[0094] Please refer to Figure 3 , Figure 3 A schematic flow chart of the detailed steps for determining appearance features involved in the video understanding method provided in an embodiment of the present application.

[0095] like Figure 3As shown, in some embodiments, the above step S101, determining the apparent features of each frame of the video to be understood at multiple spatial scales through the spatial pyramid model, may include steps S301 to S303.

[0096] Step S301, performing multi-layer convolution processing on each frame image of the video to be understood by using the spatial pyramid model to obtain multi-layer convolution features of each frame image;

[0097] After the terminal device directly inputs the video to be understood into the spatial pyramid model, the spatial pyramid model uses multiple convolution layers to perform multi-layer convolution processing on each frame of the video to be understood, thereby obtaining multi-layer convolution features of each frame of the video to be understood.

[0098] For example, it is assumed that when the terminal device constructs a spatial pyramid model using multi-level information of a convolutional neural network, it uses a deep residual network ResNet-34 as a feature extraction backbone network to learn video features. The deep residual network ResNet-34 includes 5 convolution stages, wherein the first convolution layer Conv1 uses a 7×7 convolution kernel to process the input image, and each convolution stage from the second convolution layer Conv2 to the fifth convolution layer Conv5 is composed of multiple residual blocks, each residual block includes a convolution layer and a nonlinear activation layer to process the features output by the previous convolution layer, and the input and output of the residual block are connected to form a residual connection to accelerate network optimization. In this way, after the terminal device inputs the video to be understood into the spatial pyramid model, the spatial pyramid model performs 5 convolution stages in a bottom-up order for each frame of the video to be understood, and uses the features output by each convolution layer after one convolution stage of processing the image as a layer of convolution features of the image, so that multi-layer convolution features of each frame of the video to be understood are obtained.

[0099] Step S302, determining C target layer convolution features from the multi-layer convolution features in the order of the convolution layers in the spatial pyramid model from high to low; C is a positive integer greater than 1, and C is less than the number of convolution layers in the spatial pyramid model;

[0100] After the terminal device obtains the multi-layer convolution features of each frame image in the video to be understood through the spatial pyramid model, the spatial pyramid model selects C target layer convolution features from the multi-layer convolution features of each frame image in the descending order of the convolution layers for convolution processing of each frame image.

[0101] In some embodiments, the terminal device may select each layer of convolutional features from the multi-layer convolutional features of each frame of image through the spatial pyramid model in descending order of the convolutional layers as the target layer convolutional features. For example, when the spatial pyramid model processes each frame of image through five convolutional stages to obtain five layers of convolutional features for each frame of image, for each frame of image, the terminal device may sequentially determine the five layers of convolutional features as the target layer convolutional features in descending order of the convolutional layers, thereby obtaining five target convolutional features.

[0102] In other embodiments, the terminal device may also select at least two levels of convolution features as target layer convolution features from the multi-layer convolution features of each frame of the video to be understood in the order of convolution layers from high to low through the spatial pyramid model. For example, when the spatial pyramid model processes each frame of the image in five convolution stages to obtain five layers of convolution features for each frame of the image, for each frame of the image, the terminal device may determine the fifth layer of the highest layer, the fourth layer of the middle layer, and the third layer of the low layer as target layer convolution features in the order of convolution layers from high to low, thereby obtaining three target convolution features.

[0103] Step S303, fusing the C target layer convolution features to obtain the apparent features of each frame image at multiple spatial scales.

[0104] After selecting C target layer convolution features from the multi-layer convolution features of each frame of the video to be understood, the terminal device further connects and fuses the C target convolution features in series through a spatial pyramid model, thereby obtaining the apparent features of each frame of the video to be understood at multiple spatial scales.

[0105] Exemplarily, the terminal device inputs each frame of the video to be understood into the spatial pyramid model, and then the spatial pyramid model performs five convolution stages on each image in a bottom-up order, thereby obtaining multi-layer convolution features of each frame. Afterwards, the spatial pyramid model further selects three convolution stages from the multi-layer convolution features of each frame in the order of the convolution layers from high to low: the convolution features output by the fifth convolution layer Conv5, the fourth convolution layer Conv4, and the third convolution layer Conv3 are used as the target layer convolution features of each frame. In this way, for the target layer convolution features of each frame, the spatial pyramid model uses the convolution features output by the third convolution layer Conv3 as low-level features, the convolution features output by the fourth convolution layer Conv4 as medium-level features, and the convolution features output by the fifth convolution layer Conv5 as high-level features, so that the low-level features, medium-level features, and high-level features form a pyramid feature as the apparent features of each frame of the video to be understood at multiple spatial scales.

[0106] It should be noted that the output feature maps of the pyramid features composed of the above spatial pyramid model can be expressed as A l , A m , A h , A level ∈R C×H×W , specifically expressed as follows:

[0107] A level =[A level,1 , A level,2 ,......,A level,N ]A level,n ∈R C

[0108] Where C represents the number of channels in the feature map, H and W represent the height and width of the feature map, level∈{l,m,h} represents low, medium, and high level features, respectively, and N=HW is the number of spatial positions. In this way, the convolutional layer features are viewed as a collection of N C-dimensional position feature vectors. H, W, C, and N vary with the changes in features at different levels. For example, A l ∈R 128 ×28×28 , A m ∈R 256×14×14 , A h ∈R 512×7×7 .

[0109] In this embodiment, after the terminal device performs multi-layer convolution processing on each frame of the video to be understood through the spatial pyramid model to obtain multi-layer convolution features, it determines high, medium and low-level features from the multi-layer convolution features of each frame of the image in the order of the convolution layers from high to low in the spatial pyramid model as the target layer convolution features of each frame of the image, thereby obtaining the apparent features of each frame of the image by fusing the target layer convolution features of each frame of the image. In this way, by selecting convolution features in order from high to low and trying not to select convolution features output by the lower convolution layer, it is possible to avoid the noise information introduced by the features of the lower layers, which may affect the accuracy of the overall understanding result of the video to be understood. Moreover, not selecting the convolution features of the lower layers may also avoid the large amount of data introduced by the features of the lower layers, thereby increasing the memory usage and the amount of calculation.

[0110] The C target layer convolution features of each frame of the image determined by the terminal device through the pyramid model include top-level convolution features and non-top-level convolution features. For these two types of convolution features, the terminal device processes them in different ways and then fuses them to obtain the apparent features of each frame of the image at multiple spatial scales.

[0111] Please refer to Figure 4 , Figure 4 A schematic flow chart of the refinement steps of fusing target layer convolution features involved in the video understanding method provided in an embodiment of the present application.

[0112] like Figure 4 As shown, in some embodiments, the above-mentioned step S303, fusing the C target layer convolution features to obtain the apparent features of each frame image at multiple spatial scales, may include steps S401 to S403.

[0113] Step S401, pooling the top-level convolutional features and the non-top-level convolutional features to obtain pooled top-level convolutional features and pooled non-top-level convolutional features;

[0114] For each frame of the video to be understood, after the terminal device selects C target layer convolution features from the multi-layer convolution features through the spatial pyramid model, the terminal device further pools the top-level convolution features and non-top-level convolution features among the C target layer convolutions through the spatial pyramid model.

[0115] It should be noted that the terminal device can use the global average pooling layer through the spatial pyramid model to pool the top-level convolutional features and non-top-level convolutional features.

[0116] In some embodiments, the spatial pyramid model may pool the top-level convolutional features and non-top-level convolutional features in the C target layer convolutional features through a global average pooling layer.

[0117] In other embodiments, the spatial pyramid model can also pool the top-level convolutional features and non-top-level convolutional features at the same time through multiple global average pooling layers. It should be understood that, based on the different design needs of practical applications, the terminal device can certainly use any of the above-mentioned methods to pool the target layer convolutional features through the spatial pyramid model, or even use methods not listed here. The video understanding method provided in the embodiment of the present application does not specifically limit the structure of the global average pooling layer used by the spatial pyramid model to pool the convolutional features, and the method of using the global average pooling layer to pool the convolutional features.

[0118] Step S402, adjusting the pooled non-top convolutional features through a one-dimensional convolutional layer to obtain adjusted non-top convolutional features;

[0119] After the terminal device pools the C target layer convolutions through the spatial pyramid model, the terminal device adjusts the non-top convolution features that have been pooled using a 1×1 convolution layer through the spatial pyramid model, thereby obtaining non-top convolution features that have been pooled and further adjusted.

[0120] Exemplarily, it is assumed that the C target convolutional features of each frame image are the convolutional features output by the fifth convolutional layer, the fourth convolutional layer, and the third convolutional layer, respectively, wherein the convolutional features output by the fifth convolutional layer are top-layer convolutional features, and the convolutional features output by the fourth convolutional layer and the convolutional features output by the third convolutional layer are all non-top-layer convolutional features. After the terminal device performs global average pooling on the convolutional features output by the fifth convolutional layer, the convolutional features output by the fourth convolutional layer, and the convolutional features output by the third convolutional layer through the spatial pyramid model, the terminal device further uses a 1×1 convolutional layer through the spatial pyramid model to adjust the convolutional features output by the fourth convolutional layer and the convolutional features output by the third convolutional layer.

[0121] Step S403, fusing the pooled top-level convolutional features and the adjusted non-top-level convolutional features to obtain the apparent features of each frame of image at multiple spatial scales.

[0122] For the C target layer convolutions of each frame of image, the terminal device uses a spatial pyramid model to fuse the top-level convolution features after pooling and the non-top-level convolution features after pooling and adjustment by means of lateral connections, thereby obtaining the apparent features of each frame of image in the video to be understood at multiple spatial scales.

[0123] It should be noted that the spatial pyramid model can use a multi-level feature fusion layer to fuse the top-level convolutional features that have only been pooled and the non-top-level convolutional features that have been pooled and adjusted.

[0124] For example, please refer to Figure 5 , Figure 5 A schematic diagram of the structure of a spatial pyramid module involved in the video understanding method provided in an embodiment of the present application.

[0125] like Figure 5 As shown in the figure, after each frame of the image input to the spatial pyramid module of the terminal device is processed through 5 convolution stages, the convolution features will be further pooled into a one-dimensional feature tensor through the global average pooling layer, and finally the image classification will be completed through the linear classification layer. For example, after the terminal device inputs each frame of the video to be understood into the spatial pyramid module, the spatial pyramid module calls the pre-configured spatial pyramid model, and first processes each frame of the image through 5 convolution stages in a bottom-up order from the first convolution layer Conv1 to the fifth convolution layer Conv5, and obtains the multi-layer convolution features of each frame of the image. Then, the spatial pyramid model selects the convolution features output by the fifth convolution layer Conv5, the fourth convolution layer Conv4, and the third convolution layer Conv3 from the multi-layer convolution features of each frame image in the order of convolution layers from high to low as the three target layer convolution features of each frame image, and uses the global average pooling layer to pool the convolution features output by the fifth convolution layer Conv5, the fourth convolution layer Conv4, and the third convolution layer Conv3 of each frame image into a one-dimensional feature tensor. Then, the spatial pyramid model further uses a 1×1 convolution layer to adjust the convolution features output by the fourth convolution layer of each frame image, and uses a 1×1 convolution layer to adjust the convolution features output by the third convolution layer of each frame image. Finally, the spatial pyramid model uses a multi-level feature fusion layer to connect in series the convolutional features of the three target layers of each frame image, only the convolutional features output by the fifth convolutional layer after pooling, and the convolutional features output by the fourth convolutional layer Conv4 and the third convolutional layer Conv3 after pooling and adjustment, thereby fusing the high-level features output by the fifth convolutional layer Conv5, the middle-level features output by the fourth convolutional layer, and the low-level features output by the third convolutional layer, to obtain a pyramid feature with rich semantics at multiple spatial scales for each frame of image, which represents the apparent features of the content in each frame of image at multiple spatial scales.

[0126] In this embodiment, the terminal device globally averages the convolution features of different layers along the spatial dimension through the spatial pyramid model to obtain one-dimensional feature tensors of different lengths, and then adjusts the low-level features and middle-level features after pooling through the spatial pyramid model using a 1×1 convolution layer. Finally, the high-level features after pooling and the low-level features and middle-level features after pooling and adjustment are connected in series through the spatial pyramid, thereby obtaining the apparent features of each frame of the video to be understood at multiple spatial scales. In this way, the inherent multi-level advantages of the convolutional neural network features can be fully utilized through the spatial pyramid model, and the features of the multiple spatial scale information contained in each frame of the video to be understood can be learned while only introducing a small amount of computational loss and parameters, thereby forming the pyramid features of each frame of the video to be understood, and then the pyramid features are used as the apparent features of each frame of the video to be understood for behavioral temporal reasoning, which can improve the accuracy of understanding the video to be understood.

[0127] After the terminal device processes each frame of the video to be understood through the spatial pyramid model to obtain the apparent features of each frame, it further processes these apparent features through the temporal pyramid model. The temporal pyramid model mainly includes a frame pooling layer and a relationship reasoning layer at different time intervals. It groups and processes the apparent features of each frame of the video to be understood at the video clip level, thereby inferring the temporal relationships of multiple behaviors of the video to be understood.

[0128] Please refer to Figure 6 , Figure 6 A schematic flow chart of the refinement steps for processing appearance features involved in the video understanding method provided in an embodiment of the present application.

[0129] like Figure 6 As shown, in some embodiments, the above-mentioned step S102, through the temporal pyramid model, groups the apparent features of at least part of the images in the video to be understood at the multiple spatial scales to obtain multiple behavioral temporal relationships of the video to be understood, which may include steps S601 to S603.

[0130] Step S601, dividing at least part of the images in the video to be understood into multiple groups of video segments;

[0131] When the terminal device performs grouping processing on the apparent features of at least part of the images in the video to be understood by using the temporal pyramid model, the terminal device first divides all or part of the image frames of the video to be understood into multiple groups of video segments at equal intervals by using the temporal pyramid model, wherein each group of video segments contains the same number of image frames.

[0132] Step S602, pooling the apparent features of the images contained in each of the multiple groups of video clips through the frame pooling layer to obtain multiple video clip features; the apparent features contained in the multiple groups of video clips are the apparent features of the image frames included in each of the multiple groups of video clips.

[0133] After the terminal device divides the video to be understood into multiple groups of video clips through the temporal pyramid model, it further uses the frame pooling layers corresponding to each group of video clips through the temporal pyramid model to pool the apparent features of the images included in the multiple groups of video clips, thereby obtaining the video clip features corresponding to the multiple groups of video clips.

[0134] In some embodiments, the terminal device may further process each frame of the video to be understood in the spatial pyramid model to obtain the apparent features of each frame, and then group the video frame feature sequences composed of the apparent features of each frame to obtain multiple groups of video clips. Then, the terminal device respectively pools the apparent features contained in the multiple groups of video clips through the frame pooling layers at different time intervals in the temporal pyramid model to obtain multiple video clip features.

[0135] It should be noted that the terminal device may also divide the video frame feature sequence into multiple groups of video segments at equal intervals, so that the multiple groups of video segments also have the same time range.

[0136] For example, please refer to Figure 7 , Figure 7 A schematic diagram of a scene of frame timing pooling involved in the video understanding method provided in an embodiment of the present application.

[0137] like Figure 7 As shown in the figure, it is assumed that the video to be understood contains a total of 12 image frames. After the terminal device processes each image frame through the spatial pyramid model to obtain the apparent features of each image frame at multiple spatial scales, the terminal device combines the apparent features of the 12 image frames into a video frame feature sequence, and divides the video frame feature sequence into 4 groups of non-overlapping 3-frame video clips at equal intervals according to the time interval of each group of 3 frames. Then, the terminal device uses the 3-frame relationship pooling layer corresponding to the number of frames 3 in each group of video frames through the temporal pyramid model to average pool the apparent features contained in the 3 frames of images in each group of video clips, thereby obtaining 4 video clip features.

[0138] The specific formula for the temporal pyramid model to average pool the apparent features contained in each group of video clips is as follows:

[0139]

[0140] in, Indicates the apparent features contained in the i-th frame image, represents the K-frame pooling operation, Represents the features of the sth group of video clips after K-frame pooling operation.

[0141] In some embodiments, the number of frame pooling layers included in the temporal pyramid model is multiple, and the time intervals of the multiple frame pooling layers are different. For example, in addition to the above-mentioned 3-frame relationship pooling layers, the frame pooling layers included in the temporal pyramid model also include 2-frame relationship pooling layers, 4-frame relationship pooling layers, and so on. Among them, the time interval of the 3-frame relationship pooling layer is 3 image frames, and the time intervals of the 2-frame relationship pooling layer and the 4-frame relationship pooling layer are 2 image frames and 4 image frames, respectively. In addition, the time ranges included in each of the multiple groups of video clips are the same. For example, after the above-mentioned terminal device divides the video frame feature sequence composed of the apparent features of each of the 12 image frames into 4 groups of non-overlapping 3-frame video clips at equal intervals according to each group of 3 frames, the time range included in each group of video clips is 3 image frames.

[0142] Based on this, the above-mentioned pooling of the apparent features of the images contained in each of the multiple groups of video clips through the frame pooling layer may include:

[0143] Calling a target pooling layer in the pooling layer based on the time range; a time interval of the target pooling layer corresponds to the time range;

[0144] The target pooling layer is used to pool the apparent features of the images contained in each of the multiple groups of video clips.

[0145] When the terminal device uses the frame pooling layer to pool the apparent features of the images contained in each of the multiple video clips through the temporal pyramid model, the terminal device first selects a target pooling layer corresponding to the time range among the frame pooling layers of multiple different time intervals according to the time range of the multiple video clips through the temporal pyramid model. Then, the terminal device uses the target pooling layer to pool the apparent features of the images contained in each of the multiple video clips through the temporal pyramid model.

[0146] Exemplarily, after the terminal device divides the video frame feature sequence composed of the apparent features of each of the 12 image frames into 4 groups of non-overlapping 3-frame video clips at equal intervals, the temporal pyramid model selects a 3-frame relationship pooling layer with a time interval of 3 image frames from multiple frame pooling layers such as a 2-frame relationship pooling layer, a 3-frame relationship pooling layer, and a 4-frame relationship pooling layer according to the time range of the 3-frame video clip: 3 image frames. Then, the temporal pyramid model pools the apparent features contained in each group of 3-frame video clips through the 3-frame relationship pooling layer, thereby obtaining video clip features.

[0147] Step S603: performing relational reasoning on the multiple video clip features through the relational reasoning layer to obtain multiple behavior temporal relationships.

[0148] After the terminal device uses the frame pooling layer to group and pool the appearance features through the temporal pyramid model to obtain multiple video clip features, the terminal device further uses the relational reasoning layer to perform relational reasoning on the multiple video clip features through the temporal pyramid model, thereby obtaining the behavior temporal relationships of multiple different time scales of the video to be understood. Then, the terminal device can fuse these behavior temporal relationships of different time scales in a horizontal connection manner, thereby obtaining a temporal structure in which the behavior in the video to be understood is represented by multiple time scales, and this temporal structure represents the understanding result obtained by understanding the video to be understood.

[0149] In some embodiments, the terminal device can directly perform relational reasoning on each video clip feature through the temporal pyramid model, thereby obtaining the behavior temporal relationship corresponding to each video clip feature, and then the terminal device fuses the behavior temporal relationships corresponding to the video clip features with the same time intervals through the temporal pyramid model as a behavior temporal relationship of one time scale. In this way, the terminal device divides the video frame feature sequence by using different time ranges each time, and pools the apparent features contained in the multiple groups of video clips obtained by the division to obtain multiple video clip features, and performs relational reasoning on the multiple video clip features, and then fuses the reasoning to obtain the behavior temporal relationship, so as to obtain the behavior temporal relationship of multiple different time scales of the entire video to be understood.

[0150] In other embodiments, the terminal device may also first connect the video clip features with the same time interval in series through the temporal pyramid model, and then perform relational reasoning on the connected features, so as to directly obtain a behavior temporal relationship of a time scale. In this way, the terminal device divides the video frame feature sequence by using different time ranges each time, and pools the apparent features contained in the multiple groups of video clips obtained by the division to obtain multiple video clip features, and connects the video clip features with the same time interval in the multiple video clip features in series, and then performs relational reasoning on the connected features, so as to obtain the behavior temporal relationship of multiple different time scales of the entire video to be understood.

[0151] Based on this, the above step S603, performing relational reasoning on the multiple video clip features through the relational reasoning layer to obtain multiple behavior temporal relationships, may include:

[0152] Connecting video segment features with the same time interval in series among the multiple video segment features to obtain multiple series features;

[0153] The relational reasoning layer performs relational reasoning on the multiple series features respectively to obtain multiple behavior temporal relationships.

[0154] The terminal device divides the video frame feature sequence using different time ranges through the temporal pyramid model, thereby obtaining multiple groups of video clips in different time ranges. Afterwards, after the terminal device pools the apparent features contained in each of the multiple groups of video clips obtained through the temporal pyramid model to obtain multiple video clip features, it further connects the video clip features with the same time interval among the multiple video clip features through the temporal pyramid model to obtain multiple serial features. In this way, the terminal device uses the relational reasoning layer through the temporal pyramid model to perform relational reasoning on the multiple serial features, and can obtain the behavioral temporal relationships of multiple different time scales of the video to be understood.

[0155] It should be noted that the time intervals of the multiple serial features are different. In addition, the number of relational reasoning layers included in the temporal pyramid model can also be multiple, and the time intervals corresponding to the multiple relational reasoning layers are different. For example, the relational reasoning layers included in the temporal pyramid model may include: 2-frame relational reasoning layers, 3-frame relational reasoning layers, and 4-frame relational reasoning layers, etc. Among them, the time interval corresponding to the 2-frame relational reasoning layer is 2 image frames, the time interval corresponding to the 3-frame relational reasoning layer is 3 image frames, and the time interval corresponding to the 4-frame relational reasoning layer is 4 image frames.

[0156] Based on this, the above-mentioned performing relational reasoning on the multiple series features respectively through the relational reasoning layer may include:

[0157] Relational reasoning is performed on a target serial feature among the multiple serial features through a target serial reasoning layer among the multiple serial reasoning layers.

[0158] Among them, the time interval corresponding to the target relationship reasoning layer is the same as the time interval of the target series features.

[0159] When a terminal device performs relational reasoning on multiple serial features through the time series pyramid model, for any target serial feature among the multiple serial features, the terminal device first determines a target relational reasoning layer from multiple relational reasoning layers through the time series pyramid model. The time interval corresponding to the target relational reasoning layer is the same as the time interval of the target serial feature. Then, the terminal device uses the target relational reasoning layer to perform relational reasoning on the target serial feature through the time series pyramid model.

[0160] For example, please refer to Figure 8 , Figure 8 A schematic diagram of the structure of a temporal pyramid module involved in a video understanding method provided in an embodiment of the present application.

[0161] like Figure 8 As shown, the terminal device divides the video frame feature sequence composed of the apparent features of each of the 12 image frames into 4 groups of non-overlapping 3-frame video clips according to each group of 3 frames, and calls the pre-configured temporal pyramid model through the temporal pyramid module, and uses the 3-frame relationship pooling layer to pool the apparent features contained in each group of 3-frame video clips, thereby obtaining 4 video clip features. After that, the temporal pyramid model connects the 4 video clip features in series to obtain a series feature with a time interval of 3 image frames. After that, the terminal device divides the video frame feature sequence into 6 groups of non-overlapping 2-frame video clips according to each group of 2 frames, and uses the 2-frame relationship pooling layer through the temporal pyramid model to pool the apparent features contained in each group of 2-frame video clips, thereby obtaining 6 video clip features. The terminal device also further connects the 6 video clip features in series through the temporal pyramid model to obtain a series feature with a time interval of 2 image frames. Similarly, the terminal device can also obtain a series feature with a time interval of 4 image frames through the temporal pyramid model.

[0162] Afterwards, for the serial features with a time interval of 3 image frames, the terminal device uses the 3-frame relationship reasoning layer among the 2-frame relationship reasoning layer, the 3-frame relationship reasoning layer and the 4-frame relationship reasoning layer as the target relationship reasoning through the temporal pyramid model, and uses the 3-frame relationship reasoning layer to reason for the serial features with a time interval of 3 image frames, thereby obtaining the behavior temporal relationship with a time scale of 3 image frames. In addition, for the serial features with a time interval of 2 image frames, the terminal device uses the 2-frame relationship reasoning layer to reason through the temporal pyramid model, thereby obtaining the behavior temporal relationship with a time scale of 2 image frames, and for the serial features with a time interval of 4 image frames, the terminal device uses the 4-frame relationship reasoning layer to reason through the temporal pyramid model, thereby obtaining the behavior temporal relationship with a time scale of 4 image frames.

[0163] It should be noted that when the terminal device uses the relational reasoning layer to infer the serial features, it can specifically use two layers of fully connected layers for temporal relational reasoning. Taking the above-mentioned temporal pyramid model using the 3-frame relational reasoning layer to perform relational reasoning on the serial features with a time interval of 3 image frames as an example, the terminal device uses the 3-frame relational reasoning layer to concatenate 4 video clip features (the time interval of each video clip feature is 3 image frames) through the temporal pyramid model, and uses two layers of fully connected layers for temporal relational reasoning to learn the behavioral temporal features under the 3-frame relationship from the 4 video clips:

[0164] , .

[0165] The symbols [,] represent the serial operation. and represents the parameter matrix of the K-frame relationship inference layer, Represents the result of concatenating features of multiple K video clips (the concatenated features with a time interval of K image frames), Represents the behavior timing relationship of K frames (the time scale is the behavior timing relationship of K image frames).

[0166] In addition, the 2-frame relationship pooling layer and the 2-frame relationship reasoning layer are used to learn the 2-frame temporal relationship of the behavior, and the 4-frame relationship pooling layer and the 5-frame relationship reasoning layer are used to learn the 4-frame temporal relationship of the behavior. Thus, the terminal device can model the temporal structure of the behavior from different time scales through the temporal pyramid model.

[0167] Finally, in order to fully represent the temporal relationship of the behavior, the terminal device fuses different frame relationships through the temporal pyramid model to obtain the behavioral temporal feature representation of the video to be understood at multiple time scales:

[0168] R p =R2+R3+...+R M .

[0169] The subscripts {2, 3, ..., M} of R represent different relational reasoning layers contained in the temporal pyramid model. In the temporal pyramid model, each relational reasoning layer has independent parameters, and the parameters of each relational reasoning layer are updated according to the behavior classification loss. p Represent the temporal structure of the behavior in the video to be understood at multiple time scales.

[0170] In this embodiment, the terminal device divides the video frame feature sequence using different time ranges through the temporal pyramid model, thereby obtaining multiple groups of video clips in different time ranges. After that, the terminal device pools the apparent features contained in each of the multiple groups of video clips obtained through the temporal pyramid model to obtain multiple video clip features. After that, the terminal device further connects the video clip features with the same time interval in the multiple video clip features through the temporal pyramid model to obtain multiple serial features. Thus, the terminal device uses the temporal pyramid model to use the relational reasoning layer to perform relational reasoning on the multiple serial features respectively, and can obtain the behavior temporal relationships of multiple different time scales of the video to be understood. Finally, the terminal device fuses these behavior temporal relationships of different time scales in a horizontal connection manner, thereby obtaining a temporal structure in which the behavior in the video to be understood is represented by multiple time scales, and the temporal structure represents the understanding result obtained by understanding the video to be understood.

[0171] That is to say, the terminal device repeatedly divides the video image feature sequence through the temporal pyramid model to form multiple groups of video clips with different time ranges, and then infers the behavioral temporal relationship of the video to be understood at multiple time scales from the perspective of the video clips. In this way, the accuracy of the video understanding result can be ensured, and the amount of calculation can be reduced, thereby improving the efficiency of video understanding.

[0172] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiment of the present application also provides a video understanding device for implementing the video understanding method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more video understanding device embodiments provided below can refer to the limitations of the video understanding method above, and will not be repeated here.

[0174] In one embodiment, Fig. 9 As shown, a video understanding device is provided, including: a spatial pyramid module 901, a temporal pyramid module 902 and a fusion module 903, wherein:

[0175] A spatial pyramid module 901 is used to determine the apparent features of each frame of the video to be understood at multiple spatial scales through a spatial pyramid model;

[0176] A temporal pyramid module 902 is used to group and process the apparent features of at least part of the images in the video to be understood at the multiple spatial scales through a temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood; the multiple behavior temporal relationships have different time scales;

[0177] The fusion module 903 is used to fuse the multiple behavior temporal relationships to obtain the understanding result of the video to be understood.

[0178] In some embodiments, the spatial pyramid module 901 includes:

[0179] A convolution unit, used for performing multi-layer convolution processing on each frame image of the video to be understood through a spatial pyramid model to obtain multi-layer convolution features of each frame image;

[0180] A convolution feature selection unit, configured to determine C target layer convolution features from the multi-layer convolution features in the descending order of the convolution layers in the spatial pyramid model; C is a positive integer greater than 1, and C is less than the number of convolution layers in the spatial pyramid model;

[0181] The convolution feature fusion unit is used to fuse the C target layer convolution features to obtain the apparent features of each frame image at multiple spatial scales.

[0182] In some embodiments, the C target layer convolutional features include: top-layer convolutional features and non-top-layer convolutional features;

[0183] The convolution feature fusion unit is also used to pool the top-level convolution features and the non-top-level convolution features to obtain pooled top-level convolution features and pooled non-top-level convolution features; adjust the pooled non-top-level convolution features through a one-dimensional convolution layer to obtain adjusted non-top-level convolution features; and fuse the pooled top-level convolution features and the adjusted non-top-level convolution features to obtain the apparent features of each frame of the image at multiple spatial scales.

[0184] In some of these embodiments, the temporal pyramid model includes: a frame pooling layer and a relational reasoning layer;

[0185] The time pyramid module 902 includes:

[0186] A feature grouping unit, configured to divide at least part of the images in the video to be understood into a plurality of groups of video segments;

[0187] A pooling unit, configured to pool the apparent features of the images respectively contained in the plurality of groups of video clips through the frame pooling layer to obtain a plurality of video clip features; the apparent features contained in the plurality of groups of video clips are the apparent features of the image frames respectively included in the plurality of groups of video clips;

[0188] The reasoning unit is used to perform relational reasoning on the multiple video clip features through the relational reasoning layer to obtain multiple behavior temporal relationships.

[0189] In some embodiments, the number of the frame pooling layers is multiple, and the time intervals of the multiple frame pooling layers are different; the time ranges of the multiple groups of video clips are the same;

[0190] The pooling unit is also used to call a target pooling layer in the pooling layer based on the time range; the time interval of the target pooling layer corresponds to the time range; and, through the target pooling layer, respectively pool the apparent features of the images contained in each of the multiple groups of video clips.

[0191] In some of the embodiments, the reasoning unit is further used to concatenate video clip features with the same time interval among the multiple video clip features to obtain multiple concatenated features; and, through the relationship reasoning layer, perform relationship reasoning on the multiple concatenated features respectively to obtain multiple behavior temporal relationships.

[0192] In some embodiments, the number of the relational reasoning layers is multiple, and the time intervals corresponding to the multiple relational reasoning layers are different; the time intervals of the multiple series features are different;

[0193] The reasoning unit is further configured to perform relational reasoning on a target series feature among the multiple series features through a target series reasoning layer among the multiple series reasoning layers;

[0194] The time interval corresponding to the target relationship reasoning layer is the same as the time interval of the target series features.

[0195] Each module in the above terminal device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0196] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store model parameters related to the liquid metal reactor. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video understanding method is implemented.

[0197] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0198] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0199] Determine the apparent features of each frame of the video to be understood at multiple spatial scales through a spatial pyramid model;

[0200] The appearance features of at least part of the images in the video to be understood at the multiple spatial scales are grouped and processed by a temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood; the multiple behavior temporal relationships have different time scales;

[0201] The multiple behavior temporal relationships are integrated to obtain an understanding result of the video to be understood.

[0202] In some embodiments, when the processor executes the computer program, the processor further implements the following steps:

[0203] Performing multi-layer convolution processing on each frame image of the video to be understood through a spatial pyramid model to obtain multi-layer convolution features of each frame image;

[0204] Determine C target layer convolution features from the multi-layer convolution features in the order of the convolution layers in the spatial pyramid model from high to low; C is a positive integer greater than 1, and C is less than the number of convolution layers in the spatial pyramid model;

[0205] The C target layer convolutional features are fused to obtain the apparent features of each frame of the image at multiple spatial scales.

[0206] In some embodiments, the C target layer convolution features include: top layer convolution features and non-top layer convolution features; when the processor executes the computer program, the following steps are also implemented:

[0207] Pooling the top-level convolutional features and the non-top-level convolutional features to obtain pooled top-level convolutional features and pooled non-top-level convolutional features;

[0208] Adjusting the pooled non-top convolution features through a one-dimensional convolution layer to obtain adjusted non-top convolution features;

[0209] The pooled top-level convolutional features and the adjusted non-top-level convolutional features are fused to obtain the apparent features of each frame of the image at multiple spatial scales.

[0210] In some embodiments, the temporal pyramid model includes: a frame pooling layer and a relationship reasoning layer; when the processor executes the computer program, the following steps are also implemented:

[0211] Dividing at least part of the images in the video to be understood into multiple groups of video segments;

[0212] The frame pooling layer is used to pool the apparent features of the images contained in the multiple groups of video clips respectively, so as to obtain multiple video clip features; the apparent features contained in the multiple groups of video clips are the apparent features of the image frames contained in the multiple groups of video clips respectively;

[0213] The relational reasoning layer performs relational reasoning on the multiple video clip features to obtain multiple behavior temporal relationships.

[0214] In some embodiments, the number of the frame pooling layers is multiple, and the time intervals of the multiple frame pooling layers are different; the time ranges of the multiple groups of video clips are the same; and when the processor executes the computer program, the following steps are further implemented:

[0215] Calling a target pooling layer in the pooling layer based on the time range; a time interval of the target pooling layer corresponds to the time range;

[0216] The target pooling layer is used to pool the apparent features of the images contained in each of the multiple groups of video clips.

[0217] In some embodiments, when the processor executes the computer program, the processor further implements the following steps:

[0218] Connecting video segment features with the same time interval in series among the multiple video segment features to obtain multiple series features;

[0219] The relational reasoning layer performs relational reasoning on the multiple series features respectively to obtain multiple behavior temporal relationships.

[0220] In some embodiments, the number of the relationship reasoning layers is multiple, and the time intervals corresponding to the multiple relationship reasoning layers are different; the time intervals of the multiple series features are different; when the processor executes the computer program, the following steps are also implemented:

[0221] Performing relational reasoning on a target series feature among the multiple series features through a target series reasoning layer among the multiple series reasoning layers;

[0222] The time interval corresponding to the target relationship reasoning layer is the same as the time interval of the target series features.

[0223] In one embodiment, a computer program product is provided, including a computer program. The steps implemented when the computer program is executed by a processor are the same as the steps implemented when the processor in the above-mentioned computer device executes the computer program, and the same contents will not be repeated here.

[0224] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0225] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0226] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video understanding method, characterized in that: The method comprises: Determine the apparent features of each frame of the video to be understood at multiple spatial scales by using a spatial pyramid model; wherein the spatial pyramid model can be constructed using the inherent multi-level information of a convolutional neural network, and the apparent features at multiple spatial scales are pyramid features of multiple spatial scales obtained by fusing feature representations of different levels of the video to be understood by the spatial pyramid model using lateral connections; The apparent features of at least part of the images in the video to be understood at the multiple spatial scales are grouped and processed by a temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood, and the multiple behavior temporal relationships have different time scales; wherein the at least part of the images include: half of all the images in the video to be understood extracted by using an alternate frame processing method, and the grouping processing includes: group pooling; The multiple behavior temporal relationships are integrated to obtain an understanding result of the video to be understood.

2. The method according to claim 1, characterized in that The method of determining the apparent features of each frame of the video to be understood at multiple spatial scales by using a spatial pyramid model includes: Performing multi-layer convolution processing on each frame image of the video to be understood through a spatial pyramid model to obtain multi-layer convolution features of each frame image; Determine C target layer convolution features from the multi-layer convolution features in the order of the convolution layers in the spatial pyramid model from high to low; C is a positive integer greater than 1, and C is less than the number of convolution layers in the spatial pyramid model; The C target layer convolutional features are fused to obtain the apparent features of each frame of the image at multiple spatial scales.

3. The method according to claim 2, characterized in that The C target layer convolution features include: top layer convolution features and non-top layer convolution features; The fusing of the C target layer convolution features to obtain the apparent features of each frame of image at multiple spatial scales includes: Pooling the top-level convolutional features and the non-top-level convolutional features to obtain pooled top-level convolutional features and pooled non-top-level convolutional features; Adjusting the pooled non-top convolution features through a one-dimensional convolution layer to obtain adjusted non-top convolution features; The pooled top-level convolutional features and the adjusted non-top-level convolutional features are fused to obtain the apparent features of each frame of the image at multiple spatial scales.

4. The method according to any one of claims 1 to 3, characterized in that The temporal pyramid model includes: a frame pooling layer and a relationship reasoning layer; The method of grouping the apparent features of at least part of the images in the video to be understood at the multiple spatial scales by using the temporal pyramid model to obtain multiple temporal relationships of behaviors in the video to be understood includes: Dividing at least part of the images in the video to be understood into multiple groups of video segments; The frame pooling layer is used to pool the apparent features of the images contained in the multiple groups of video clips respectively, so as to obtain multiple video clip features; the apparent features contained in the multiple groups of video clips are the apparent features of the image frames contained in the multiple groups of video clips respectively; The relational reasoning layer performs relational reasoning on the multiple video clip features to obtain multiple behavior temporal relationships.

5. The method according to claim 4, characterized in that The number of the frame pooling layers is multiple, and the time intervals of the multiple frame pooling layers are different; the time ranges of the multiple groups of video clips are the same; The step of pooling the apparent features of the images respectively contained in the plurality of groups of video clips through the frame pooling layer comprises: Calling a target pooling layer in the pooling layer based on the time range; a time interval of the target pooling layer corresponds to the time range; The target pooling layer is used to pool the apparent features of the images contained in each of the multiple groups of video clips.

6. The method according to claim 4, characterized in that The performing relational reasoning on the multiple video clip features through the relational reasoning layer to obtain multiple behavior temporal relationships includes: Connecting video segment features with the same time interval in series among the multiple video segment features to obtain multiple series features; The relational reasoning layer performs relational reasoning on the multiple series features respectively to obtain multiple behavior temporal relationships.

7. The method according to claim 6, characterized in that The number of the relational reasoning layers is multiple, and the time intervals corresponding to the multiple relational reasoning layers are different; the time intervals of the multiple series features are different; The performing relational reasoning on the plurality of series features respectively through the relational reasoning layer includes: Performing relational reasoning on a target series feature among the multiple series features through a target series reasoning layer among the multiple series reasoning layers; The time interval corresponding to the target relationship reasoning layer is the same as the time interval of the target series features.

8. A video understanding device, characterized in that: The device comprises: A spatial pyramid module is used to determine the apparent features of each frame of the video to be understood at multiple spatial scales through a spatial pyramid model; wherein the spatial pyramid model can be constructed using the inherent multi-level information of a convolutional neural network, and the apparent features at multiple spatial scales are pyramid features of multiple spatial scales obtained by the spatial pyramid model using lateral connections to fuse feature representations of different levels of the video to be understood; A temporal pyramid module is used to group the apparent features of at least part of the images in the video to be understood at the multiple spatial scales through a temporal pyramid model to obtain multiple behavior temporal relationships of the video to be understood; the multiple behavior temporal relationships have different time scales; wherein the at least part of the images include: half of all the images in the video to be understood extracted by using an alternate frame processing method, and the grouping processing includes: group pooling; The fusion module is used to fuse the multiple behavior temporal relationships to obtain the understanding result of the video to be understood.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the video understanding method as described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video understanding method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Video motion recognition method based on fusion of sorting pooling and spatial features

    CN108416795A

  • Behavior recognition method, device and equipment and storage medium

    CN112580523A

  • Video behavior recognition method, system, device and equipment

    CN115512267A

  • Behavior recognition method, device and equipment and computer readable storage medium

    CN116152904A

  • Work violation identification method combined with multi-spatial scale features

    CN118334747A