Video analysis method and device, computer equipment and storage medium

Abstract

The invention relates to a video analysis method and device, computer equipment and a storage medium, and the method comprises the steps: transmitting a to-be-analyzed video through 3D convolution, so as to obtain a plurality of non-overlapping first image feature blocks; obtaining a second image feature block through a heat conduction operator model according to the first image feature block; screening out a third image feature block from the second image feature block through a time dimension; obtaining a fourth image feature block from the third image feature block through a heat conduction operator model; screening a fifth image feature block from the fourth image feature block through a spatial dimension; and acquiring a sixth image feature block from the fifth image feature block through the heat conduction operator model, and taking the sixth image feature block as a target feature block of the to-be-analyzed video. According to the invention, the processing efficiency of video processing can be improved during video analysis.

Description

Technical Field

[0001] The present invention relates to the field of video analysis, and in particular to a video analysis method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of information technology and the Internet, the demand for video analysis is increasing. Video analysis generally refers to analyzing required information from videos.

[0003] In the field of video analysis, commonly used technologies can be mainly divided into the following categories: First, methods based on traditional feature extraction, which mostly rely on manually designed features, such as optical flow (Optical Flow), HOG (Histogram of Oriented Gradients), SIFT (Scale-Invariant Feature Transform), etc. Although the computational complexity is low, it is difficult to handle complex scenes and the efficiency is also low.

[0004] The second approach is spatial feature extraction based on convolutional neural networks (CNNs). This method uses a 2D CNN model to extract features from single-frame images and capture spatial information. While this method effectively extracts spatial information from each frame, it ignores temporal dynamics and struggles to capture the continuity of actions or events. If dynamic information needs to be processed, spatial feature extraction based on CNNs becomes extremely inefficient.

[0005] It can be seen that the video analysis methods in the prior art are difficult to achieve high-efficiency analysis while meeting user needs. Summary of the Invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a video analysis method, apparatus, computer equipment and storage medium.

[0007] In a first aspect, the present invention provides a video analysis method, the method comprising:

[0008] Transmitting the video to be analyzed using 3D convolution to obtain a plurality of non-overlapping first image feature blocks;

[0009] According to the first image feature block, obtaining a second image feature block through a heat conduction operator model;

[0010] Filtering a third image feature block from the second image feature block according to a time dimension;

[0011] Obtaining a fourth image feature block from the third image feature block through a heat conduction operator model;

[0012] Filtering a fifth image feature block from the fourth image feature block according to a spatial dimension;

[0013] A sixth image feature block is obtained from the fifth image feature block through the heat conduction operator model, and the sixth image feature block is used as a target feature block of the video to be analyzed.

[0014] Optionally, the video to be analyzed is transmitted using 3D convolution to obtain multiple non-overlapping first image feature blocks in the following manner:

[0015]

[0016] N=t×h×w

[0017] t=T

[0018]

[0019] w=W / 16

[0020] in, is the first image feature block, N is the number of first image feature blocks, C is the number of channels of the first image feature block, T is the number of RGB image frames contained in the video to be analyzed, H and W are the number of row pixels and column pixels of the RGB image respectively.

[0021] Optionally, obtaining a second image feature block through a heat conduction operator model according to the first image feature block includes:

[0022] Obtaining an input feature block of the heat conduction operator model according to the first image feature block;

[0023] Inputting the input feature block into the heat conduction operator model to obtain the second image feature block;

[0024] The input feature block of the heat conduction operator model is obtained according to the first image feature block in the following manner:

[0025]

[0026] Among them, X 0 is the input feature block of the heat conduction operator model, is the first image feature block, p s is the embedded spatial location information, p t is the embedded spatiotemporal location information.

[0027] Optionally, the heat conduction operator model includes L HCOLayer layers, each of which includes two LN units, an HCO Block unit, and an FFN unit. The HCO Block unit includes a depth convolution layer, a first linear layer, a second linear layer, a third linear layer, an activation function layer, an HCO operator, and an LN unit.

[0028] The input feature block is input into the heat conduction operator model to obtain the second image feature block in the following manner:

[0029] X′=DWConv(LN(z l-1 ))

[0030] X 0 =Linear2(X′),

[0031] HCOBlock(LN(z l-1 ))

[0032] =Linear3(SiLU(Linear1(X′))⊙LN(HCO(X 0 ))))

[0033] z′ l =z l-1 +HCOBlock(LN(z l-1 ))

[0034] HCOLayer(z l-1 )=zl l +FFN(LN(z′ l ))

[0035] z l =HCOLayer(z l-1 )

[0036] Y=head(z L )

[0037] l=1…L

[0038] Among them, X ′ is the first intermediate value, DWConv means passing through the depth convolution layer, LN means passing through the LN unit, HCOBlock means after passing through the HCOBlock unit, FFN means after passing through the FFN unit, Linear1 means passing through the first linear layer, Linear2 means passing through the second linear layer, HCOLayer means passing through the HCOLayer layer, z′ l is the second intermediate value, Y is the prediction vector after classification head processing, L is the total number of HCOLayer layers, X 0is the input feature block of the heat conduction operator model, SiLU is the activation function, z L It is the second image feature block output by the last layer, and HCO represents the calculation by the HCO operator;

[0039] Calculated by the HCO operator, as follows:

[0040]

[0041] Where t is the diffusion time, X t is the image feature after diffusion time t, IDCT 2D is the two-dimensional discrete cosine transform, DCT 2D Two-dimensional inverse discrete cosine transform, ω x and ω y is the frequency domain coordinate, k is the adaptive thermal diffusion coefficient obtained by embedding the frequency value into dynamic prediction, and N is the number of feature blocks of the video image.

[0042] Optionally, filtering out the third image feature block from the second image feature block according to the time dimension includes:

[0043] Performing average pooling on the second image feature block along the spatial dimension to obtain a first temporal feature block;

[0044] Inputting the first time feature block into a lightweight Scorer network to predict a first importance score of each of the first time feature blocks;

[0045] According to the first importance score of each first time feature block, a preset number of the first importance scores are selected from high to low as screening scores, and the first time feature blocks corresponding to the screening scores are used as the third image feature blocks.

[0046] Optionally, the second image feature block is average pooled along the spatial dimension to obtain the first temporal feature block in the following manner:

[0047] q t =AvgPool space (z l )∈R t×C

[0048] z l is the second image feature block, q t is the first time feature block;

[0049] The first time feature block is input into the lightweight Scorer network to predict the first importance score of each first time feature block in the following manner:

[0050] f l=FC(q;θ1)∈R L×C′

[0051] f g =AvgPool(f l )∈R C′

[0052]

[0053] C′=C / 2

[0054]

[0055] s′=FC(f;θ2)∈R N×1

[0056]

[0057] Among them, f l is the local feature, FC represents the fully connected layer, q is the first time feature block, θ1 is the parameter of the first fully connected layer, f g is the global feature, AvgPool represents the average pooling layer, f i The local feature f l With the global feature f g The concatenated feature block, s is the first importance score, s′ is the third intermediate parameter, and θ2 is the parameter of the second fully connected layer FC.

[0058] Optionally, filtering out the fifth image feature block from the fourth image feature block according to the spatial dimension includes:

[0059] Inputting the fourth image feature block into a lightweight Scorer network to predict a second importance score of each of the fourth image feature blocks;

[0060] constructing the second importance score into a two-dimensional score map;

[0061] dividing overlapping anchor points on the two-dimensional score map;

[0062] Obtaining the average score of the overlapping anchor points, and taking the overlapping anchor point with the highest average score as the retained anchor point;

[0063] All the fourth image feature blocks in the area corresponding to the retained anchor point are used as the fifth image feature blocks.

[0064] In a second aspect, a video analysis device is provided, the device comprising:

[0065] A convolution model, configured to transmit the video to be analyzed using 3D convolution to obtain a plurality of non-overlapping first image feature blocks;

[0066] A heat conduction operator model, configured to obtain a second image feature block based on the first image feature block;

[0067] A time dimension screening model, configured to screen out a third image feature block from the second image feature block based on the time dimension;

[0068] The heat conduction operator model is further used to obtain a fourth image feature block from the third image feature block;

[0069] A spatial dimension screening model, configured to screen out a fifth image feature block from the fourth image feature block based on a spatial dimension;

[0070] The heat conduction operator model is further used to obtain a sixth image feature block from the fifth image feature block, and use the sixth image feature block as a target feature block of the video to be analyzed.

[0071] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.

[0072] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method as described in any one of the above items is implemented.

[0073] The present invention provides a video analysis method, apparatus, computer device, and storage medium. The method comprises: using 3D convolution to convolve a video to be analyzed to obtain multiple non-overlapping first image feature blocks; obtaining a second image feature block based on the first image feature blocks using a heat conduction operator model; filtering a third image feature block from the second image feature block using the time dimension; obtaining a fourth image feature block from the third image feature block using the heat conduction operator model; filtering a fifth image feature block from the fourth image feature block using the space dimension; and obtaining a sixth image feature block from the fifth image feature block using the heat conduction operator model, with the sixth image feature block serving as the target feature block of the video to be analyzed. The heat conduction operator model utilizes global modeling capabilities while achieving efficient computation, significantly improving video processing efficiency. The present invention also significantly reduces inefficient computation and memory overhead by filtering in both the time and space dimensions, resulting in faster processing speeds. Furthermore, the present invention alternates between the heat conduction operator model and the filtering in both the time and space dimensions, achieving a balanced approach to processing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0076] Figure 1 FIG2 is an application environment diagram of the video analysis method according to an embodiment of the present invention;

[0077] Figure 2 FIG2 is a flow chart of a video analysis method according to an embodiment of the present invention;

[0078] Figure 3 FIG2 is a schematic diagram of a heat conduction operator model according to an embodiment of the present invention;

[0079] Figure 4 FIG2 is a flow chart of a video analysis method according to an embodiment of the present invention;

[0080] Figure 5 FIG2 is a structural block diagram of a video analysis device according to an embodiment of the present invention;

[0081] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0082] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0083] Figure 1 FIG. 1 is an application environment diagram of a video analysis method in an embodiment. Figure 1 The video analysis method is applied to a video analysis system. The video analysis method includes a terminal 110 and / or a server 120. Terminal 110 and server 120 are connected via a network. Terminal 110 can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, and a laptop computer. Server 120 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0084] The video analysis method of the present invention is applied to the terminal 110 and / or the server 120 .

[0085] Figure 2 Shown is a flow chart of the video analysis method of the present invention, as shown in Figure 2 As shown, the method includes:

[0086] Step 210, using 3D convolution to transmit the video to be analyzed to obtain a plurality of non-overlapping first image feature blocks;

[0087] Step 220: Obtain a second image feature block using a heat conduction operator model based on the first image feature block.

[0088] Step 230: Filter out a third image feature block from the second image feature block based on the time dimension;

[0089] Step 240: Obtain a fourth image feature block from the third image feature block using a heat conduction operator model;

[0090] Step 250: Filtering a fifth image feature block from the fourth image feature block by spatial dimension;

[0091] Step 260: Obtain a sixth image feature block from the fifth image feature block through the heat conduction operator model, and use the sixth image feature block as the target feature block of the video to be analyzed.

[0092] In this embodiment of the present invention, a heat conduction operator model is employed, which provides global modeling capabilities while achieving efficient computation, significantly improving video processing efficiency. This invention also significantly reduces inefficient computation and memory overhead by filtering in both the temporal and spatial dimensions, resulting in faster processing speeds. Furthermore, the present invention alternates between the heat conduction operator model and the temporal and spatial dimension filtering, achieving a balanced approach to processing efficiency and accuracy.

[0093] In an embodiment of the present invention, in step 210, the video to be analyzed is projected using 3D convolution to obtain a plurality of non-overlapping first image feature blocks in the following manner:

[0094]

[0095] N=t×h×w

[0096] t=T

[0097]

[0098] w=W / 16

[0099] in, is the first image feature block, N is the number of first image feature blocks, C is the number of channels of the first image feature block, T is the number of RGB image frames contained in the video to be analyzed, H and W are the number of row pixels and column pixels of the RGB image respectively.

[0100] In the embodiments of the present invention, 3D convolution is used to directly process the video to be analyzed, which can increase temporal and spatial information, help extract dynamic feature blocks between consecutive frames, and simultaneously retain temporal and positional information, thereby improving the perception of the timing and spatial structure of actions in the method of the present application. In addition, convolution can effectively extract dynamic features in consecutive frames, improving the subsequent model's ability to understand the timing and spatial structure of actions.

[0101] In the embodiment of the present invention, step 210, obtaining the second image feature block by using a heat conduction operator model based on the first image feature block, includes:

[0102] Obtaining an input feature block of the heat conduction operator model according to the first image feature block;

[0103] Inputting the input feature block into the heat conduction operator model to obtain the second image feature block;

[0104] The input feature block of the heat conduction operator model is obtained according to the first image feature block in the following manner:

[0105]

[0106] Among them, X 0 is the input feature block of the heat conduction operator model, is the first image feature block, p s is the embedded spatial location information, p t is the embedded spatiotemporal location information.

[0107] In an embodiment of the present invention, the heat conduction operator model includes L HCOLayer layers, each of which includes two LN units, an HCO Block unit, and an FFN unit. The HCO Block unit includes a depth convolution layer, a first linear layer, a second linear layer, a third linear layer, an activation function layer, an HCO operator, and an LN unit.

[0108] The input feature block is input into the heat conduction operator model to obtain the second image feature block in the following manner:

[0109] X′=DWConv(LN(z l-1 ))

[0110] X 0 =Linear2(X′),

[0111] HCOBlock(LN(z l-1 ))

[0112] =Linear3(SiLU(Linear1(X′))⊙LN(HCO(X 0 ))))

[0113] z′ l =z l-1 +HCOBlock(LN(z l-1 ))

[0114] HCOLayer(z l-1 )=Z′ l +FFN(LN(z′ l ))

[0115] z l =HCOLayer(z l-1 )

[0116] Y=head(z L )

[0117] l=1…L

[0118] Where x′ is the first intermediate value, DWConv means passing through the depth convolution layer, LN means passing through the LN unit, HCOBlock means after passing through the HCOBlock unit, FFN means after passing through the FFN unit, Linear1 means passing through the first linear layer, Linear2 means passing through the second linear layer, HCOLayer means passing through the HCOLayer layer, and z′ l is the second intermediate value, Y is the prediction vector after classification head processing, L is the total number of HCOLayer layers, X 0 is the input feature block of the heat conduction operator model, SiLU is the activation function, z L It is the second image feature block output by the last layer, and HCO represents the calculation by the HCO operator.

[0119] Figure 3 The figure shows a schematic diagram of the heat conduction operator model of an embodiment of the present invention, and the connection mode of each layer and each functional unit is as follows: Figure 3 As shown, in other embodiments of the present invention, the connection method of each layer and each functional unit can also be other methods, which will not be repeated here.

[0120] In the embodiment of the present invention, the calculation is performed by the HCO operator in the following manner:

[0121]

[0122] Where t is the diffusion time, X t is the image feature after diffusion time t, IDCT 2D is the two-dimensional discrete cosine transform, DCT 2DTwo-dimensional inverse discrete cosine transform, ω x and ω y is the frequency domain coordinate, k is the adaptive thermal diffusion coefficient obtained by embedding the frequency value into dynamic prediction, and N is the number of feature blocks of the video image.

[0123] In the embodiment of the present invention, step 230, filtering out the third image feature block from the second image feature block based on the time dimension, includes:

[0124] Performing average pooling on the second image feature block along the spatial dimension to obtain a first temporal feature block;

[0125] Inputting the first time feature block into a lightweight Scorer network to predict a first importance score of each of the first time feature blocks;

[0126] According to the first importance score of each first time feature block, a preset number of the first importance scores are selected from high to low as screening scores, and the first time feature blocks corresponding to the screening scores are used as the third image feature blocks.

[0127] In the embodiment of the present invention, the second image feature block is average pooled along the spatial dimension to obtain the first temporal feature block in the following manner:

[0128] q t =AvgPool space (z l )∈R t×C

[0129] z l is the second image feature block, q t is the first time feature block;

[0130] The first time feature block is input into the lightweight Scorer network to predict the first importance score of each first time feature block in the following manner:

[0131] f l =FC(q;θ1)∈R L×C′

[0132] f g =AvgPool(f l )∈R C′

[0133]

[0134] C′=C / 2

[0135]

[0136] s′=FC(f;θ2)∈R N×1

[0137]

[0138] Among them, f l is the local feature, FC represents the fully connected layer, q is the first time feature block, θ1 is the parameter of the first fully connected layer, f g is the global feature, AvgPool represents the average pooling layer, f i The local feature f l With the global feature f g The concatenated feature block, s is the first importance score, s′ is the third intermediate parameter, and θ2 is the parameter of the second fully connected layer FC.

[0139] Since the computational overhead of the lightweight scorer network is extremely low, in the embodiments of the present invention, through the screening of the time dimension and the design of the lightweight scorer network, high accuracy and low overhead can be maintained simultaneously, balancing efficiency and performance.

[0140] In the embodiment of the present invention, filtering out the fifth image feature block from the fourth image feature block according to the spatial dimension includes:

[0141] Inputting the fourth image feature block into a lightweight Scorer network to predict a second importance score of each of the fourth image feature blocks;

[0142] constructing the second importance score into a two-dimensional score map;

[0143] dividing overlapping anchor points on the two-dimensional score map;

[0144] Obtaining the average score of the overlapping anchor points, and taking the overlapping anchor point with the highest average score as the retained anchor point;

[0145] All the fourth image feature blocks in the area corresponding to the retained anchor point are used as the fifth image feature blocks.

[0146] In the embodiment of the present invention, although spatial and temporal screening have been performed, the key timing and spatial information in the video are still reasonably retained. At the same time, spatial and temporal screening have been performed, which effectively reduces redundant calculations. Therefore, the present invention can significantly reduce the computing cost without losing accuracy, and significantly improves the reasoning efficiency of the model.

[0147] Figure 4 FIG. 1 is a flow chart of a video analysis method according to an embodiment of the present invention. Figure 4 The flowchart shown is based on the model structure.

[0148] like Figure 4 As shown in the figure, after the video to be analyzed is convolved, it passes through several HCOLayer layers of the heat conduction operator model, and then undergoes time filtering, continues to pass through several HCOLayer layers, and then undergoes spatial filtering, passes through several HCOLayer layers again, and then is output.

[0149] Figure 4 The several HCOLayer layers shown may be layers in one heat conduction operator model, or may belong to different heat conduction operator models.

[0150] Figure 4 In the illustrated embodiment, the placement of the temporal and spatial filters can be adjusted based on the video being analyzed and the desired results. The closer the temporal and spatial filters are to the video being analyzed, the lower the accuracy, but the faster the calculation speed. The closer the temporal and spatial filters are to the output, the higher the accuracy, but the slower the calculation speed.

[0151] In the method of this embodiment, in order to balance accuracy and efficiency, time screening can be set at the 6th to 12th HCOLayer layers, and spatial screening can be performed at intervals of 2 HCOLayer layers after time screening.

[0152] like Figure 5 As shown, the present invention also provides a video analysis device, the device comprising:

[0153] A convolution model 510 is configured to transmit the video to be analyzed using 3D convolution to obtain a plurality of non-overlapping first image feature blocks;

[0154] a heat conduction operator model 520, configured to obtain a second image feature block based on the first image feature block;

[0155] A time dimension screening model 530 is configured to screen out a third image feature block from the second image feature block based on the time dimension;

[0156] The heat conduction operator model 520 is further configured to obtain a fourth image feature block from the third image feature block;

[0157] A spatial dimension screening model 540 is configured to screen out a fifth image feature block from the fourth image feature block based on a spatial dimension;

[0158] The heat conduction operator model 520 is further configured to obtain a sixth image feature block from the fifth image feature block, and use the sixth image feature block as a target feature block of the video to be analyzed.

[0159] In an embodiment of the present invention, the convolution model 510 is further configured to project the video to be analyzed using 3D convolution in the following manner to obtain a plurality of non-overlapping first image feature blocks:

[0160]

[0161] N=t×h×w

[0162] t=T

[0163]

[0164] w=W / 16

[0165] in, is the first image feature block, N is the number of first image feature blocks, C is the number of channels of the first image feature block, T is the number of RGB image frames contained in the video to be analyzed, H and W are the number of row pixels and column pixels of the RGB image respectively.

[0166] In the embodiment of the present invention, the heat conduction operator model 520 is further used to:

[0167] Obtaining an input feature block of the heat conduction operator model according to the first image feature block;

[0168] Inputting the input feature block into the heat conduction operator model to obtain the second image feature block;

[0169] The heat conduction operator model is further configured to obtain an input feature block of the heat conduction operator model according to the first image feature block in the following manner:

[0170] X 0 =χ V +p s +p t

[0171] Among them, X 0 is the input feature block of the heat conduction operator model, is the first image feature block, p s is the embedded spatial location information, p t is the embedded spatiotemporal location information.

[0172] The heat conduction operator model includes L HCOLayer layers, each of which includes two LN units, an HCO Block unit and an FFN unit. The HCO Block unit includes a depth convolution layer, a first linear layer, a second linear layer, a third linear layer, an activation function layer, an HCO operator and an LN unit.

[0173] The heat conduction operator model 520 is further configured to input the input feature block into the heat conduction operator model in the following manner to obtain the second image feature block:

[0174] X′=DWConv(LN(zl-1 ))

[0175] X 0 =Linear2(X′),

[0176] HCOBlock)LN(z l-1 ))

[0177] =Linear3(SiLU(Linear1(X′))⊙LN(HCO(X 0 ))))

[0178] z′ l =z l-1 +HCOBlock(LN(z l-1 ))

[0179] HCOLayer(z l-1 )=z′ l +FFN(LN(z′ l )0

[0180] z l =HCOLayer(z l-1 )

[0181] Y=head(z L )

[0182] l=1…L

[0183] Where X′ is the first intermediate value, DWConv means passing through the depth convolution layer, LN means passing through the LN unit, HCOBlock means after passing through the HCOBlock unit, FFN means after passing through the FFN unit, Linear1 means passing through the first linear layer, Linear2 means passing through the second linear layer, HCOLayer means passing through the HCOLayer layer, and z′ l is the second intermediate value, Y is the prediction vector after classification head processing, L is the total number of HCOLayer layers, X 0 is the input feature block of the heat conduction operator model, SiLU is the activation function, z L It is the second image feature block output by the last layer, and HCO represents the calculation by the HCO operator.

[0184] Calculated by the HCO operator, as follows:

[0185]

[0186] Where t is the diffusion time, t t is the image feature after diffusion time t, IDCT 2D is the two-dimensional discrete cosine transform, DCT 2DTwo-dimensional inverse discrete cosine transform, X x and ω y is the frequency domain coordinate, k is the adaptive thermal diffusion coefficient obtained by embedding the frequency value into dynamic prediction, and N is the number of feature blocks of the video image.

[0187] In the embodiment of the present invention, the time dimension screening model 530 is further used to:

[0188] Performing average pooling on the second image feature block along the spatial dimension to obtain a first temporal feature block;

[0189] Inputting the first time feature block into a lightweight Scorer network to predict a first importance score of each of the first time feature blocks;

[0190] According to the first importance score of each first time feature block, a preset number of the first importance scores are selected from high to low as screening scores, and the first time feature blocks corresponding to the screening scores are used as the third image feature blocks.

[0191] The time dimension screening model 530 is further configured to perform average pooling on the second image feature block along the spatial dimension in the following manner to obtain a first time feature block:

[0192] q t =AvgPool space (z l )∈R t×C

[0193] z l is the second image feature block, q t is the first time feature block;

[0194] The first time feature block is input into the lightweight Scorer network to predict the first importance score of each first time feature block in the following manner:

[0195] f l =FC(q;θ1)∈R L×C′

[0196] f g =AvgPool(f l )∈R C′

[0197]

[0198] C′=C / 2

[0199]

[0200] s′=FC(f;θ2)∈R N×1

[0201]

[0202] Among them, f l is the local feature, FC represents the fully connected layer, q is the first time feature block, θ1 is the parameter of the first fully connected layer, f g is the global feature, AvgPool represents the average pooling layer, f i The local feature f l With the global feature f g The concatenated feature block, s is the first importance score, s′ is the third intermediate parameter, and θ2 is the parameter of the second fully connected layer FC.

[0203] In the embodiment of the present invention, the spatial dimension screening model 540 is further used to:

[0204] Inputting the fourth image feature block into a lightweight Scorer network to predict a second importance score of each of the fourth image feature blocks;

[0205] constructing the second importance score into a two-dimensional score map;

[0206] dividing overlapping anchor points on the two-dimensional score map;

[0207] Obtaining the average score of the overlapping anchor points, and taking the overlapping anchor point with the highest average score as the retained anchor point;

[0208] All the fourth image feature blocks in the area corresponding to the retained anchor point are used as the fifth image feature blocks.

[0209] An embodiment of the present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following method is implemented: the method comprises: transmitting the video to be analyzed using 3D convolution to obtain a plurality of non-overlapping first image feature blocks; obtaining a second image feature block through a heat conduction operator model based on the first image feature block; filtering out a third image feature block from the second image feature block through a time dimension; obtaining a fourth image feature block from the third image feature block through a heat conduction operator model; filtering out a fifth image feature block from the fourth image feature block through a space dimension; obtaining a sixth image feature block from the fifth image feature block through the heat conduction operator model, and using the sixth image feature block as the target feature block of the video to be analyzed.

[0210] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following method, comprising: using 3D convolution to transmit the video to be analyzed to obtain multiple non-overlapping first image feature blocks; obtaining a second image feature block through a heat conduction operator model based on the first image feature block; filtering out a third image feature block from the second image feature block through a time dimension; obtaining a fourth image feature block from the third image feature block through a heat conduction operator model; filtering out a fifth image feature block from the fourth image feature block through a space dimension; obtaining a sixth image feature block from the fifth image feature block through the heat conduction operator model, and using the sixth image feature block as the target feature block of the video to be analyzed.

[0211] The above-mentioned video analysis method achieves the beneficial effect of being able to solve the technical problems raised in the background technology.

[0212] Figure 2 and Figure 4 FIG. 1 is a flow chart of a video analysis method in one embodiment. It should be understood that although Figure 2 and Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 and Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0213] Figure 6 The internal structure diagram of a computer device in one embodiment is shown. The computer device may be Figure 1 The server 120 in Figure 6As shown, the computer device includes a processor, a memory, a network interface, an input device, and a display screen connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the video analysis method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the video analysis method. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or a button, trackball, or touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0214] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0215] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-described methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0216] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0217] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A video analysis method, characterized in that: The method comprises: Transmitting the video to be analyzed using 3D convolution to obtain a plurality of non-overlapping first image feature blocks; According to the first image feature block, obtaining a second image feature block through a heat conduction operator model; Filtering a third image feature block from the second image feature block according to a time dimension; Obtaining a fourth image feature block from the third image feature block through a heat conduction operator model; Filtering a fifth image feature block from the fourth image feature block according to a spatial dimension; A sixth image feature block is obtained from the fifth image feature block through the heat conduction operator model, and the sixth image feature block is used as a target feature block of the video to be analyzed.

2. The method according to claim 1, characterized in that The video to be analyzed is transmitted using 3D convolution to obtain multiple non-overlapping first image feature blocks in the following manner: N=t×h×w t=T w=W / 16 in, is the first image feature block, N is the number of first image feature blocks, C is the number of channels of the first image feature block, T is the number of RGB image frames contained in the video to be analyzed, H and W are the number of row pixels and column pixels of the RGB image respectively.

3. The method according to claim 2, characterized in that The step of obtaining a second image feature block by using a heat conduction operator model according to the first image feature block includes: Obtaining an input feature block of the heat conduction operator model according to the first image feature block; Inputting the input feature block into the heat conduction operator model to obtain the second image feature block; The step of obtaining the input feature block of the heat conduction operator model according to the first image feature block is as follows: Among them, X 0 is the input feature block of the heat conduction operator model, is the first image feature block, p s is the embedded spatial location information, p t is the embedded spatiotemporal location information.

4. The method according to claim 3, characterized in that The heat conduction operator model includes L HCOLayer layers, each of which includes two LN units, an HCO Block unit and an FFN unit. The HCO Block unit includes a depth convolution layer, a first linear layer, a second linear layer, a third linear layer, an activation function layer, an HCO operator and an LN unit. The input feature block is input into the heat conduction operator model to obtain the second image feature block in the following manner: X′=DWConv(LN(z l-1 )) X 0 =Linear2(X′), HCOBlock(LN(z l-1 )) =Linear3(SiLU(Linear1(X′))⊙LN(HCO(X 0 )))) z′ l =with l-1 +HCOBlock(LN(z l-1 )) HCOLayer(z l-1 )=z′ l +FFN(LN(z′ l )) With l =HCOLayer(z l-1 ) T=head(z L ) l＝1…L Where X′ is the first intermediate value, DWConv means passing through the depth convolution layer, LN means passing through the LN unit, HCOBlock means after passing through the HCOBlock unit, FFN means after passing through the FFN unit, Linear1 means passing through the first linear layer, Linear2 means passing through the second linear layer, HCOLayer means passing through the HCOLayer layer, and z′ l is the second intermediate value, Y is the prediction vector after classification head processing, L is the total number of HCOLayer layers, X 0 is the input feature block of the heat conduction operator model, SiLU is the activation function, z L It is the second image feature block output by the last layer, and HCO represents the calculation by the HCO operator; Calculated by the HCO operator, as follows: Where t is the diffusion time, X t is the image feature after diffusion time t, IDCT 2D is the two-dimensional discrete cosine transform, DCT 2D Two-dimensional inverse discrete cosine transform, ω x and ω y is the frequency domain coordinate, k is the adaptive thermal diffusion coefficient obtained by embedding the frequency value into dynamic prediction, and N is the number of feature blocks of the video image.

5. The method according to claim 1, wherein The filtering out the third image feature block from the second image feature block according to the time dimension includes: Performing average pooling on the second image feature block along the spatial dimension to obtain a first temporal feature block; Inputting the first time feature block into a lightweight Scorer network to predict a first importance score of each of the first time feature blocks; According to the first importance score of each first time feature block, a preset number of the first importance scores are selected from high to low as screening scores, and the first time feature blocks corresponding to the screening scores are used as the third image feature blocks.

6. The method according to claim 5, characterized in that The second image feature block is average-pooled along the spatial dimension to obtain the first temporal feature block in the following manner: q t =AvgPool space (z l )∈R t×C z l is the second image feature block, q t is the first time feature block; The first time feature block is input into the lightweight Scorer network to predict the first importance score of each first time feature block in the following manner: f l =FC(q;θ1)∈R L×C′ f g =AvgPool(f l )∈R C′ C′=C / 2 s′=FC(f;θ2)∈R N×1 Among them, f l is the local feature, FC represents the fully connected layer, q is the first time feature block, θ1 is the parameter of the first fully connected layer, f g is the global feature, AvgPool represents the average pooling layer, f i The local feature f l With the global feature f g The concatenated feature block, s is the first importance score, s′ is the third intermediate parameter, and θ2 is the parameter of the second fully connected layer FC.

7. The method according to claim 1, characterized in that The step of filtering out the fifth image feature block from the fourth image feature block according to the spatial dimension includes: Inputting the fourth image feature block into a lightweight Scorer network to predict a second importance score of each of the fourth image feature blocks; constructing the second importance score into a two-dimensional score map; dividing overlapping anchor points on the two-dimensional score map; Obtaining the average score of the overlapping anchor points, and taking the overlapping anchor point with the highest average score as the retained anchor point; All the fourth image feature blocks in the area corresponding to the retained anchor point are used as the fifth image feature blocks.

8. A video analysis device, characterized in that: The device comprises: A convolution model, configured to transmit the video to be analyzed using 3D convolution to obtain a plurality of non-overlapping first image feature blocks; A heat conduction operator model, configured to obtain a second image feature block based on the first image feature block; A time dimension screening model, configured to screen out a third image feature block from the second image feature block based on the time dimension; The heat conduction operator model is further used to obtain a fourth image feature block from the third image feature block; a spatial dimension screening model, configured to screen out a fifth image feature block from the fourth image feature block according to a spatial dimension; The heat conduction operator model is further used to obtain a sixth image feature block from the fifth image feature block, and use the sixth image feature block as a target feature block of the video to be analyzed.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images