Method, system and computer readable medium for obtaining fused multi-scale first and second dimensional receptive field information

Abstract

In one embodiment, a method includes: fusing information from corresponding combinations of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field to obtain information from multi-scale first and second-dimensional receptive fields corresponding to the multi-scale second-dimensional receptive field; wherein the multi-scale first-dimensional receptive field of the first dimension is one of space and time, and the multi-scale second-dimensional receptive field of the second dimension is the other of space and time; each of the multi-scale second-dimensional receptive fields is provided by a corresponding first convolutional neural network operation set, and each operation in the first convolutional neural network operation set has a corresponding first-dimensional local-to-local constraint; and fusing the information from the combination of the multi-scale first and second-dimensional receptive fields to obtain fused multi-scale first and second-dimensional receptive field information.

Description

Technical Field

[0001] The present application relates to the field of video understanding, and in particular to a method, system and computer-readable medium for obtaining fused multi-scale first and second dimensional receptive field information. Background Art

[0002] Video is an integral part of today's big data. Driven by the progress of deep learning for images and speech, using deep learning techniques to understand video content has become one of the hottest topics. Video understanding involves analyzing video clips and interpreting what happened in them. Video classification focuses on automatically classifying / labeling video clips based on their semantic content, such as human actions or complex events. Summary of the invention

[0003] One object of the present application is to provide a method, system and computer-readable medium for obtaining fused multi-scale first and second dimensional receptive field information.

[0004] In a first aspect of the present application, a computer-implemented method includes: fusing information from corresponding combinations of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field to obtain information from multi-scale first and second-dimensional receptive fields corresponding to the multi-scale second-dimensional receptive field; wherein the multi-scale first-dimensional receptive field of the first dimension is one of space and time, and the multi-scale second-dimensional receptive field of the second dimension is the other of space and time; each of the multi-scale second-dimensional receptive fields is provided by a corresponding first convolutional neural network operation set, and each operation in the first convolutional neural network operation set has a corresponding first-dimensional local-to-local constraint; and fusing the information from the combination of the multi-scale first and second-dimensional receptive fields to obtain fused multi-scale first and second-dimensional receptive field information.

[0005] In a second aspect of the present application, a system includes at least one memory and a processor module. At least one memory is configured to store program instructions. The processor module is configured to execute program instructions, which cause the processor module to perform the following steps: fuse information from corresponding combinations of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field to obtain information from multi-scale first and second-dimensional receptive fields corresponding to the multi-scale second-dimensional receptive field; wherein the multi-scale first-dimensional receptive field of the first dimension is one of space and time, and the multi-scale second-dimensional receptive field of the second dimension is the other of space and time; each of the multi-scale second-dimensional receptive fields is provided by a corresponding first convolutional neural network operation set, and each operation in the first convolutional neural network operation set has a corresponding first-dimensional local-to-local constraint; and fuse the information from the combination of the multi-scale first and second-dimensional receptive fields to obtain fused multi-scale first and second-dimensional receptive field information.

[0006] In a third aspect of the present application, a non-temporary computer-readable medium having program instructions stored thereon is provided. When the program instructions are executed by a processor module, the processor module is caused to perform the following steps: fusing information from a corresponding combination of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field to obtain information from multi-scale first and second-dimensional receptive fields corresponding to the multi-scale second-dimensional receptive field; wherein the multi-scale first-dimensional receptive field of the first dimension is one of space and time, and the multi-scale second-dimensional receptive field of the second dimension is the other of space and time; each of the multi-scale second-dimensional receptive fields is provided by a corresponding first convolutional neural network operation set, and each operation in the first convolutional neural network operation set has a corresponding first-dimensional local-to-local constraint; and fusing the information from the combination of the multi-scale first and second-dimensional receptive fields to obtain fused multi-scale first and second-dimensional receptive field information. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the embodiments of the present application or related technologies, the following drawings that will be described in the embodiments are briefly introduced. Obviously, these drawings are only some embodiments of the present application, and ordinary technicians in this field can obtain other drawings based on these drawings without creative work.

[0008] Figure 1A and Figure 1B is a schematic diagram showing corresponding operations of a first convolutional neural network operation set, each operation in the first convolutional neural network operation set having a spatial local-to-local constraint when performed on a sequence of input feature maps to obtain temporal importance.

[0009] Figure 2A and Figure 2B is a schematic diagram showing corresponding operations of a first convolutional neural network operation set, each operation in the first convolutional neural network operation set having a temporal local-to-local constraint when performed on a sequence of input feature maps to obtain spatial importance.

[0010] Figure 3A and Figure 3B is a schematic diagram showing corresponding operations of a first convolutional neural network operation set, each operation in the first convolutional neural network operation set having a spatial local-to-local constraint when performed on a sequence of input feature maps to obtain temporal importance.

[0011] Figure 4A and Figure 4B is a schematic diagram showing corresponding operations of a first convolutional neural network operation set, each operation in the first convolutional neural network operation set having a temporal local-to-local constraint when performed on a sequence of input feature maps to obtain spatial importance.

[0012] Figure 5 is a block diagram showing input, processing, and output hardware modules in a terminal according to an embodiment of the present application.

[0013] Figure 6 is a block diagram showing a module for acquiring information of a fused multi-scale first and second dimensional receptive field according to an embodiment of the present application.

[0014] Figure 7 This is a diagram showing an embodiment of the present application. Figure 6 A block diagram of a multi-scale first and second dimensional receptive field information acquisition module in the multi-scale first and second dimensional receptive field information acquisition module includes Figure 7 Modules in .

[0015] Figure 8 is a diagram showing a combination of receptive fields according to an embodiment of the present application, the combination of receptive fields corresponding to Figure 7 The modules in the multi-scale first and second dimensional receptive field information acquisition module and Figure 6 The steps performed by the module in the fusion multi-scale first and second dimensional receptive field information acquisition module.

[0016] Fig. 9 This is a diagram showing an embodiment of the present application. Figure 7 The fusion multi-scale first-dimensional receptive field information acquisition module in Figure 7 and Figure 6 Zhongyu Fig. 9 The block diagram of the corresponding modules of the fusion multi-scale first-dimensional receptive field information acquisition module in the embodiment of the present invention includes Fig. 9 Modules in .

[0017] Fig.10 This is a diagram showing an embodiment of the present application. Figure 7 The fusion multi-scale first-dimensional receptive field information acquisition module in Figure 7 and Figure 6 Zhongyu Fig.10 The block diagram of the corresponding modules of the fusion multi-scale first-dimensional receptive field information acquisition module in the embodiment of the present invention includes Fig.10 Modules in .

[0018] Fig.11 This is a diagram showing an embodiment of the present application. Fig.10 Schematic diagram of the details of the first dimension importance acquisition module.

[0019] Fig.12 This is a diagram showing an embodiment of the present application. Fig.10A block diagram of the details of the multi-fusion important and finest first dimension scale and single second dimension scale first and second dimensional receptive field information acquisition module and the important and finest first dimension scale and important second dimension scale first and second dimensional receptive field information acquisition module.

[0020] Fig.13 is a block diagram showing a module for acquiring multi-scale first and second dimensional receptive field information fusion and multi-scale first dimensional receptive field information according to an embodiment of the present application.

[0021] Fig.14 It is a schematic diagram showing a context with at least one more suitable time scale obtained by a module for obtaining a sequence of fused time importance and input feature maps according to an embodiment of the present application.

[0022] Fig.15 It is a schematic diagram showing the background suppression effect of a fusion multi-scale first-dimensional receptive field information acquisition module according to an embodiment of the present application.

[0023] Fig.16 is a diagram showing a method according to an embodiment of the present application. Fig. 9 Schematic diagram of the context with global and finest spatial scales and important time scales obtained by the fusion multi-scale first and second dimensional receptive field information acquisition module in.

[0024] Fig.17 is a diagram showing a method according to an embodiment of the present application. Fig.10 Schematic diagram of the context with important spatial scales and important temporal scales obtained by the fusion multi-scale first and second dimensional receptive field information acquisition module in .

[0025] Fig.18 This is a diagram showing an embodiment of the present application. Figure 6 A block diagram of a multi-scale first and second dimensional receptive field information acquisition module in the multi-scale first and second dimensional receptive field information acquisition module includes Fig.18 Modules in .

[0026] Fig.19 is a diagram showing a combination of receptive fields according to an embodiment of the present application, the combination of receptive fields corresponding to Fig.18 The modules in the multi-scale first and second dimensional receptive field information acquisition module and Figure 6 The steps performed by the module in the fusion multi-scale first and second dimensional receptive field information acquisition module.

[0027] Fig. 20 is a schematic diagram showing a separable 3D Resnet-50 network having multiple fused multi-scale first and second dimensional receptive field information acquisition modules and a multi-scale first dimensional receptive field information acquisition module according to an embodiment of the present application.

[0028] Fig.21 FIG. 4 is a block diagram showing a 3D residual block according to an embodiment of the present application.

[0029] Fig. 22 FIG. 4 is a block diagram showing a 3D residual block according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical content, structural features, objectives and effects of the embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Specifically, the terms in the embodiments of the present application are only used to illustrate the purpose of specific embodiments and are not used to limit the present application.

[0031] The same reference numerals in different drawings denote substantially the same elements, and for substantially the same elements, the description of one element is applicable to the other elements.

[0032] The term "using at least one object to perform at least one operation" in this application refers to a situation where at least one object is directly used to perform at least one operation, or a situation where at least one object is modified by at least one intermediate operation and the modified at least one object is directly used to perform at least one operation.

[0033] The term "at least one object is acquired using a mechanism" in this application means that the step of acquiring at least one object includes executing the mechanism.

[0034] The term "performing at least one operation on at least one object" in the present application refers to a situation where at least one operation is performed directly using at least one object.

[0035] The term “perform at least one operation to achieve a result” in the present application refers to a case where at least one operation is performed to directly achieve a result, or a case where there is at least one intermediate operation after the at least one operation to achieve the result.

[0036] In the related art, when each frame of a video clip contains a large amount of irrelevant or unimportant information, video classification does not consider whether the information is irrelevant or unimportant, thereby affecting the classification accuracy. Some embodiments of the present application obtain the second dimension importance to force a computer vision convolutional neural network (CNN) to focus on important areas in the second dimension domain. Each operation of the first convolutional neural network operation set performed to obtain the second dimension importance has a first dimension local to local constraint. The first dimension can be one of space and time, and the second dimension can be the other of space and time.

[0037] Figure 1A and Figure 1B1 is a schematic diagram showing corresponding operations 106a and 106b of a first set of convolutional neural network operations, each of which has a spatial local-to-local constraint when executed on an input feature map sequence 102 to obtain temporal importance. The first set of convolutional neural network operations includes Figure 1A Operation 106a and Figure 1B Operation 106b in Figure 1A Operation 106a and Figure 1B Operation 106b in is performed on the input feature map sequence 102 to obtain the temporal importance. Figure 1A and Figure 1B In the example of , the size of the input feature map sequence 102 is T x Hx W, where T, H, and W represent the time, height, and width dimensions, respectively. Operations 106a and 106b of the first convolutional neural network operation set are temporal pooling operations, and the pooling window sizes of these temporal pooling operations are T / 2 and T, respectively. Each of the operations 106a and 106b of the first convolutional neural network operation set has a corresponding spatial local-to-local constraint. That is, when operation 106a of the first convolutional neural network operation set is performed on the input feature map sequence 102, due to the scale of the spatial portion of the single original scale of the spatiotemporal receptive field of operation 106a of the first convolutional neural network operation set, operation 106a of the first convolutional neural network operation set only sees the juxtaposed local spatial regions 108a and 108b in the input feature map sequence 102 at a time, and only sees the juxtaposed local spatial regions 108c and 108d in the input feature map sequence 102 at a time. When operation 106b of the first convolutional neural network operation set is performed on the input feature map sequence 102, due to the scale of the spatial portion of the single original scale of the spatiotemporal receptive field of operation 106b of the first convolutional neural network operation set, operation 106b of the first convolutional neural network operation set only sees the juxtaposed local spatial regions 108a, 108b, 108c, and 108d in the input feature map sequence 102 at a time. The local spatial regions 108a, 108b, 108c, and 108d are the corresponding original spatial regions of the input feature map sequence 102. Figure 1A and Figure 1BIn the example of , the juxtaposed local spatial regions are juxtaposed pixels. Baseball 104 flies from the center of the video frame corresponding to input feature map 102a in input feature map sequence 102 to the upper left corner of the video frame corresponding to input feature map 102d in input feature map sequence 102. Therefore, the temporal scale of the context provided by operation 106b of the first convolutional neural network operation set with a pooling window size of T is more appropriate than the temporal scale of the context provided by operation 106a of the first convolutional neural network operation set with a pooling window size of T / 2. However, due to the spatial local-to-local constraints of operation 106b of the first convolutional neural network operation set, operation 106b of the first convolutional neural network operation set is not aware of the entire flight trajectory of baseball 104 in the video frames corresponding to input feature maps 102a, 102b, 102c, and 102d. One advantage of some embodiments of the present application is that when obtaining temporal importance, context with at least one more appropriate spatiotemporal scale can be obtained, thereby improving classification accuracy. In the above example, the context with a more appropriate spatiotemporal scale may reflect the entire flight trajectory of the baseball 104 in the corresponding video frames corresponding to the input feature maps 102a, 102b, 102c, and 102.

[0038] Figure 2A and Figure 2B 206a and 206b of a first set of convolutional neural network operations, each of which has a temporal local-to-local constraint when executed on an input feature map sequence 202 to obtain spatial importance. The first set of convolutional neural network operations includes Figure 2A Operation 206a and Figure 2B Operation 206b in, Figure 2A Operation 206a and Figure 2B Operation 206b in is performed on the input feature map sequence 102 to obtain the temporal importance. Figure 2A and Figure 2BIn the example of , the size of the input feature map sequence 102 is T x H x W, where T, H, and W represent the time, height, and width dimensions, respectively. Operations 206a and 206b of the first convolutional neural network operation set are spatial pooling operations, and the pooling window sizes of these spatial pooling operations are (H / 4)x(H / 4) and (H / 2)xH / 2, respectively. Each of operations 206a and 206b of the first convolutional neural network operation set has a corresponding temporal local-to-local constraint. That is, when operation 206a of the first convolutional neural network operation set is performed on the input feature map sequence 202, due to the scale of the temporal portion of the single original scale of the spatiotemporal receptive field of operation 206a of the first convolutional neural network operation set, operation 206a of the first convolutional neural network operation set only sees the spatial area 208a corresponding to the local temporal portion 210a of the input feature map sequence 202 at a time, only sees the spatial area 208b corresponding to the local temporal portion 210b of the input feature map sequence 202 at a time, only sees the spatial area 208c corresponding to the local temporal portion 210c of the input feature map sequence 202 at a time, and only sees the spatial area 208d corresponding to the local temporal portion 210d of the input feature map sequence 202 at a time. When operation 206b of the first convolutional neural network operation set is performed on the input feature map sequence 202, due to the scale of the temporal portion of the single original scale of the spatiotemporal receptive field of operation 206b of the first convolutional neural network operation set, operation 206b of the first convolutional neural network operation set only sees the spatial region 212a corresponding to the local temporal portion 210a of the input feature map sequence 202 at one time, only sees the spatial region 212b corresponding to the local temporal portion 210b of the input feature map sequence 202 at one time, only sees the spatial region 212c corresponding to the local temporal portion 210c of the input feature map sequence 202 at one time, and only sees the spatial region 212d corresponding to the local temporal portion 210d of the input feature map sequence 202 at one time. The local temporal portions 210a, 210b, 210c, and 210d are the corresponding original temporal portions of the input feature map sequence 202. Figure 2A and Figure 2BIn the example of , each local temporal portion is of size T / T. Baseball 204 flies from the center of the video frame corresponding to input feature map 202a in input feature map sequence 202 to the upper left corner of the video frame corresponding to input feature map 202d in input feature map sequence 202. Therefore, the spatial scale of the context provided by operation 206b of the first convolutional neural network operation set with a pooling window size of (H / 2)x(H / 2) is more appropriate than the spatial scale of the context provided by operation 206a of the first convolutional neural network operation set with a pooling window size of (H / 4)x(H / 4). However, due to the temporal local-to-local constraint of operation 206b of the first convolutional neural network operation set, operation 206b of the first convolutional neural network operation set is not aware of the entire flight trajectory of baseball 204 in the video frames corresponding to input feature maps 202a, 202b, 202c, and 202d. One advantage of some embodiments of the present application is that when obtaining spatial importance, context with at least one more appropriate spatiotemporal scale can be obtained, thereby improving classification accuracy. In the above example, the context with a more appropriate spatiotemporal scale may reflect the entire flight trajectory of the baseball 204 in the corresponding video frames corresponding to the input feature maps 202a, 202b, 202c, and 202d.

[0039] Figure 3A and Figure 3B is a schematic diagram showing corresponding operations 306a and 306b of a first set of convolutional neural network operations, each of which has a spatial local-to-local constraint when performed on an input feature map sequence 102 to obtain temporal importance. Figure 1A and Figure 1B Compared to the example in Figure 3A and Figure 3B The differences between the examples in are as follows. Operations 306a and 306b of the first convolutional neural network operation set are 3D pooling operations, and the corresponding pooling window sizes of these 3D pooling operations are (T / 2)x(H / 4)x(W / 4) and (T)x(H / 4)x(W / 4), respectively. For the spatial local-to-local constraints of operation 306a of the first convolutional neural network operation set, the juxtaposed local spatial regions 308a and 308b are juxtaposed (H / 4)x(W / 4) regions, and the juxtaposed local spatial regions 308c and 308d are juxtaposed (H / 4)x(W / 4) regions. For the spatial local-to-local constraints of operation 306b of the first convolutional neural network operation set, the juxtaposed local spatial regions 308a, 308b, 308c, and 308d are juxtaposed (H / 4)x(W / 4) regions. The local spatial regions 308a, 308b, 308c and 308d are the corresponding original spatial regions of the input feature map sequence 102. Figure 1A and Figure 1BSimilar to the example in , the time scale of the context provided by operation 306b of the first convolutional neural network operation set with a pooling window size of T is more appropriate than the time scale of the context provided by operation 306a of the first convolutional neural network operation set with a pooling window size of T / 2. However, due to the spatial local-to-local constraints of operation 306b of the first convolutional neural network operation set, operation 306b of the first convolutional neural network operation set is not aware of the entire flight trajectory of baseball 104 in the video frames corresponding to input feature maps 102a, 102b, 102c, and 102d. One advantage of some embodiments of the present application is that context with at least one more appropriate spatiotemporal scale can be obtained when obtaining temporal importance, thereby improving classification accuracy. In the above example, the context with a more appropriate spatiotemporal scale can reflect the entire flight trajectory of baseball 104 in the corresponding video frames corresponding to input feature maps 102a, 102b, 102c, and 102d.

[0040] Figure 4A and Figure 4B 4 is a schematic diagram showing corresponding operations 406a and 406b of a first set of convolutional neural network operations, each of which has a temporal local-to-local constraint when performed on the input feature map sequence 202 to obtain spatial importance. Figure 2A and Figure 2B Compared to the example in . Figure 4A and Figure 4B The differences between the examples in are as follows. Operations 406a and 406b of the first convolutional neural network operation set are 3D pooling operations, and the pooling window sizes of these 3D pooling operations are (T / 2)x(H / 4)x(W / 4) and (T / 2)x(H / 2)x(W / 2), respectively. For the temporal local-to-local constraint of operation 406a of the first convolutional neural network operation set, the size of the local temporal portion 410a of the input feature map sequence 202 is T / 2, and the size of the local temporal portion 410b of the input feature map sequence 202 is T / 2. For the temporal local-to-local constraint of operation 406b of the first convolutional neural network operation set, the size of the local temporal portion 410a of the input feature map sequence 202 is T / 2, and the size of the local temporal portion 410b of the input feature map sequence 202 is T / 2. The local temporal portions 410a, 410b, 410c, and 410d are the corresponding original temporal portions of the input feature map sequence 202. Figure 2A and Figure 2BSimilar to the example in , the spatial scale of the context provided by operation 406b of the first convolutional neural network operation set with a pooling window size of (H / 2)x(H / 2) is more appropriate than the spatial scale of the context provided by operation 406a of the first convolutional neural network operation set with a pooling window size of (H / 4)x(H / 4). However, due to the temporal local-to-local constraint of operation 406b of the first convolutional neural network operation set, operation 406b of the first convolutional neural network operation set is not aware of the entire flight trajectory of baseball 204 in the video frames corresponding to input feature maps 202a, 202b, 202c, and 202d. One advantage of some embodiments of the present application is that when obtaining spatial importance, context with at least one more appropriate spatiotemporal scale can be obtained, thereby improving classification accuracy. In the above example, the context with a more appropriate spatiotemporal scale can reflect the entire flight trajectory of baseball 204 in the corresponding video frames corresponding to input feature maps 202a, 202b, 202c, and 202d.

[0041] Figure 5 is a block diagram showing input, processing and output hardware modules in a terminal 500 according to an embodiment of the present application. Figure 5 As shown, terminal 500 includes a camera module 502, a processor module 504, a memory module 506, a display module 508, a storage module 510, a wired or wireless communication module 512, and a bus 514. Terminal 500 can be a mobile phone, a smart phone, a tablet computer, a laptop computer, a desktop computer, or any electronic device with sufficient computing power to perform video understanding.

[0042] The camera module 502 is an input hardware module for collecting video clips 2002 (exemplarily marked in Fig. 20 ), the video clip 2002 is transmitted to the processor module 504 via the bus 514. The camera module 502 includes an RGB (Red-Green-Blue) camera or a grayscale camera. Alternatively, the video clip 2002 can be obtained using other input hardware modules, such as the storage module 510, or the wired or wireless communication module 512. The storage module 510 is used to store the video clip 2002, and the video clip 2002 is transmitted to the processor module 504 via the bus 514. The wired or wireless communication module 512 is used to receive the video clip 2002 from the network via wired or wireless communication, and the video clip 2002 is transmitted to the processor module 504 via the bus 514.

[0043] The memory module 506 stores the inference phase program instructions, and the processor module 504 executes the inference phase program instructions, so that the processor module 504 implements the computer vision CNN (for example, the separable 3D Resnet-50 network 2000, the separable 3DResnet-50 network 2000 has multiple fusion multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information acquisition modules G1 to G2, and H1 to H3, which will refer to Fig. 20 ) to process the video clips. The memory module 506 may be a transient or non-transient computer-readable medium including at least one memory. The processor module 504 includes at least one processor that directly or indirectly sends signals to and / or receives signals from the camera module 502, the memory module 506, the display module 508, the storage module 510, and the wired or wireless communication module 512 via the bus 514. The at least one processor may be a central processing unit (CPU), a graphics processing unit (GPU), and / or a digital signal processor (DSP). The CPU may send the facial image 508, some program instructions, and other data or instructions to the GPU and / or the DSP via the bus 514.

[0044] The display module 508 is an output hardware module for displaying the results obtained by the computer vision CNN. Alternatively, the results obtained by the computer vision CNN can be output through other output hardware modules, such as the storage module 510, or the wired or wireless communication module 512. The storage module 510 is used to store the results obtained by the computer vision CNN. The wired or wireless communication module 512 is used to transmit the results obtained by the computer vision CNN to the network via wired or wireless communication. The output hardware module receives the results obtained by the computer vision CNN from the processor module 504 via the bus 514.

[0045] In the above embodiment, terminal 500 is a computing system, all components of which are integrated together via bus 514. Other types of computing systems, such as computing systems with remote camera modules instead of camera module 502, are within the scope of the present application.

[0046] Figure 6is a block diagram showing a fused multi-scale first and second dimensional receptive field information acquisition module 600 according to an embodiment of the present application. By applying the fused multi-scale first and second dimensional receptive field information acquisition module 600 to a computer vision CNN, the computer vision CNN focuses on the first dimensional context enhancement important area in the second dimensional domain. In the present application, the first dimension may be one of space and time, and the second dimension may be the other of space and time. The fused multi-scale first and second dimensional receptive field information acquisition module 600 is configured to acquire fused multi-scale first and second dimensional receptive field information 610 using an input feature map sequence 606. The fused multi-scale first and second dimensional receptive field information acquisition module 600 includes a multi-scale first and second dimensional receptive field information acquisition module 602 and a multi-scale first and second dimensional receptive field information fusion module 604.

[0047] The multi-scale first and second dimensional receptive field information acquisition module 602 is configured to fuse information from a corresponding combination of all multi-scale first dimensional receptive fields and each multi-scale second dimensional receptive field using the input feature map sequence 606 to obtain information from the multi-scale first and second dimensional receptive fields corresponding to the multi-scale second dimensional receptive field 608. The multi-scale first and second dimensional receptive field information acquisition module 602 performs a corresponding first convolutional neural network operation set that provides each multi-scale second dimensional receptive field. Each operation of the first convolutional neural network operation set has a corresponding first dimensional local-to-local constraint.

[0048] The multi-scale first and second dimensional receptive field information fusion module 604 is configured to fuse information 608 from a combination of multi-scale first and second dimensional receptive fields to obtain fused multi-scale first and second dimensional receptive field information 610 .

[0049] Figure 7 This is a diagram showing an embodiment of the present application. Figure 6 The multi-scale first and second dimensional receptive field information acquisition module 602 includes Figure 7 The multi-scale first and second dimensional receptive field information acquisition module 602 includes a fusion multi-scale first dimensional receptive field information acquisition module 702 and a multi-fusion multi-first dimensional scale and single second dimensional scale first and second dimensional receptive field information acquisition module 704.

[0050] The fused multi-scale first-dimensional receptive field information acquisition module 702 is configured to use the input feature map sequence 606 to perform at least one second convolutional neural network operation to obtain fused multi-scale first-dimensional receptive field information 706. At least one of the multi-scale first-dimensional receptive fields is provided by the corresponding at least one second convolutional neural network operation, wherein the scale of at least one of the multi-scale first-dimensional receptive fields is greater than the minimum scale of the multi-scale first-dimensional receptive field.

[0051] The multiple fused multiple first-dimension scales and single second-dimension scale first and second-dimension receptive field information acquisition module 704 is configured to perform each of the multiple third convolutional neural network operations using the fused multi-scale first-dimension receptive field information 706 to obtain corresponding fused multiple first-dimension scales and single second-dimension scale first and second-dimension receptive field information 708. Each set of first convolutional neural network operations is a corresponding third convolutional neural network operation that provides a corresponding multi-scale second-dimension receptive field. The corresponding fused multiple first-dimension scales and single second-dimension scale first and second-dimension receptive field information 708 of all third convolutional neural network operations is information 608 from the multi-scale first and second-dimension receptive field.

[0052] Figure 8 is a schematic diagram showing a combination of receptive fields according to an embodiment of the present application, the combination of receptive fields corresponding to Figure 7 Modules 702 and 704 in the multi-scale first and second dimensional receptive field information acquisition module 602 and Figure 6 The steps performed by module 604 in the fusion multi-scale first and second dimensional receptive field information acquisition module 600. Figure 8 In the example of , for the fused multi-scale first-dimensional receptive field information acquisition module 702, at least one second convolutional neural network operation is performed. Exemplarily, the at least one second convolutional neural network operation includes at least one of the multi-scale first-dimensional receptive fields 802 and 804. Fused multi-scale first-dimensional receptive field information 706 is acquired. That is, information from the combination 810 of the multi-scale first-dimensional receptive fields 802 and 804 is fused.

[0053] For the multi-fused multi-first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 704, each of the third convolutional neural network operations is performed. Exemplarily, the third convolutional neural network operation includes two operations, which have multi-scale second-dimension receptive fields 806 and 808 respectively. Each of the third convolutional neural network operations is performed using the fused multi-scale first-dimension receptive field information 706 to obtain corresponding fused multi-first-dimension scale and single second-dimension scale first and second-dimension receptive field information 708. That is, information from the corresponding combination 812 or 814 of all multi-scale first-dimension receptive fields 802 and 804 and each of the multi-scale second-dimension receptive fields 806 and 808 is fused. The corresponding fused multi-first-dimension scale and single second-dimension scale first and second-dimension receptive field information 708 of all third convolutional neural network operations is information 608 from the multi-scale first and second-dimension receptive fields. That is, the information 608 from the multi-scale first and second dimensional receptive fields is information from a combination 812 of all the multi-scale first dimensional receptive fields 802 and 804 and one 806 of the multi-scale second dimensional receptive fields 806 and 808, and a combination 814 of all the multi-scale first dimensional receptive fields 802 and 804 and the other 808 of the multi-scale second dimensional receptive fields 806 and 808.

[0054] For the multi-scale first and second dimensional receptive field information fusion module 604, information 608 from the combination of the multi-scale first and second dimensional receptive fields is fused to obtain fused multi-scale first and second dimensional receptive field information 610. That is, information from the combination 812 of all the multi-scale first dimensional receptive fields 802 and 804 and one 806 of the multi-scale second dimensional receptive fields 806 and 808 and the combination 814 of all the multi-scale first dimensional receptive fields 802 and 804 and the other 808 of the multi-scale second dimensional receptive fields 806 and 808 is fused.

[0055] Fig. 9 This is a diagram showing an embodiment of the present application. Figure 7 The fusion multi-scale first-dimensional receptive field information acquisition module 702 in , and Figure 7 and Figure 6 Zhongyu Fig. 9 The fusion multi-scale first-dimensional receptive field information acquisition module 702 in the block diagram of the corresponding modules 704 and 604, the fusion multi-scale first-dimensional receptive field information acquisition module 702 includes Fig. 9 Modules 902 and 904 in.

[0056] The fused multi-scale first-dimensional receptive field information acquisition module 702 includes a global first-dimensional information acquisition module 902 and a global first-dimensional information and input feature map sequence fusion module 904 .

[0057] The global first dimension information acquisition module 902 is configured to use the input feature map sequence 606 to perform at least one second convolutional neural network operation to obtain the global first dimension information 910. Figure 8 As shown, at least one second convolutional neural network operation has only one operation, which has one 802 of the multi-scale first-dimensional receptive fields 802 and 804. One 802 of the multi-scale first-dimensional receptive fields 802 and 804 has a global scale.

[0058] The global first dimension information and input feature map sequence fusion module 904 is configured to fuse the global first dimension information 910 and the input feature map sequence 606 to obtain fused global and finest scale first dimension receptive field information 912. In one embodiment, the fusion performed by the global first dimension information and input feature map sequence fusion module 904 is an element-by-element summation operation. Figure 8 As shown, the input feature map sequence 606 is obtained by an operation having another 804 of the multi-scale first-dimensional receptive fields 802 and 804. Because the other 804 of the multi-scale first-dimensional receptive fields 802 and 804 is directly the scale of the first-dimensional receptive field of the operation that obtains the input feature map sequence 606, rather than the scale of the first-dimensional receptive field of one of the at least one second convolutional neural network operations that takes the input feature map sequence 606 as input, the other 804 of the multi-scale first-dimensional receptive fields 802 and 804 has the finest scale. By fusing the global first-dimensional information 910 and the input feature map sequence 606, information from the combination 810 of the multi-scale first-dimensional receptive fields 802 and 804 is fused.

[0059] like Fig. 9 As shown, accordingly, the multi-fused multi-first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 704 includes a multi-fused global and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 906. The multi-fused global and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 906 is configured to perform each third convolutional neural network operation using the fused global and finest scale first-dimension receptive field information 912 to obtain the corresponding fused global and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information 914. The corresponding fused global and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information 914 of all third convolutional neural network operations is the information 608 from the multi-scale first and second-dimension receptive fields.

[0060] Accordingly, the multi-scale first and second dimensional receptive field information fusion module 604 includes a global and finest first dimension scale and important second dimension scale first and second dimensional receptive field information fusion module 908. The global and finest first dimension scale and important second dimension scale first and second dimensional receptive field information fusion module 908 is configured to fuse the information from the multi-scale first and second dimensional receptive field 608 to obtain the global and finest first dimension scale and important second dimension scale first and second dimensional receptive field information 916.

[0061] Figure 8 The corresponding combination of the receptive fields in the example is applicable to the present embodiment after necessary modifications, Figure 8 The corresponding combination of receptive fields in the Figure 7 The multi-fusion multi-first dimension scale and single second dimension scale first and second dimension receptive field information acquisition module 704 and Figure 6 The corresponding steps performed by the multi-scale first and second dimensional receptive field information fusion module 604.

[0062] Fig.10 This is a diagram showing an embodiment of the present application. Figure 7 The fusion multi-scale first-dimensional receptive field information acquisition module 702 in , and Figure 7 and Figure 6 Zhongyu Fig.10 The fusion multi-scale first-dimensional receptive field information acquisition module 702 in the block diagram of the corresponding modules 704 and 604, the fusion multi-scale first-dimensional receptive field information acquisition module 702 includes Fig.10 Modules 1002 and 1004 in.

[0063] The fused multi-scale first-dimension receptive field information acquisition module 702 includes a first-dimension importance acquisition module 1002 and a first-dimension importance and input feature map sequence fusion module 1004 .

[0064] The first dimension importance acquisition module 1002 is configured to use the input feature map sequence 606 to perform at least one second convolutional neural network operation to obtain the first dimension importance 1010. Figure 8 As shown, illustratively, at least one second convolutional neural network operation has two operations, and the two operations respectively have one 802 of the multi-scale first-dimensional receptive fields 802, 804, and 818 (not shown) and another 804 of the multi-scale first-dimensional receptive fields 802, 804, and 818 (not shown). One 802 of the multi-scale first-dimensional receptive fields 802, 804, and 818 (not shown) has a global scale. Another 804 of the multi-scale first-dimensional receptive fields 802, 804, and 818 (not shown) has a scale finer than the global scale. Other numbers of second convolutional neural network operations with corresponding first-dimensional receptive fields of different corresponding scales are within the scope of this application.

[0065] The first dimension importance and input feature map sequence fusion module 1004 is configured to fuse the first dimension importance 1010 and the input feature map sequence 606 to obtain fused important and finest scale first dimension receptive field information 1012. In one embodiment, the fusion performed by the first dimension importance and input feature map sequence fusion module 1004 is an element-by-element summation operation. The first dimension importance 1010 is added to the input feature map sequence 606 as residual information. Figure 8 As shown, the input feature map sequence 606 is obtained by an operation having a multi-scale first-dimensional receptive field 802, 804, and 818 (not shown). Because the multi-scale first-dimensional receptive field 802, 804, and 818 (not shown) is directly the scale of the first-dimensional receptive field of the operation to obtain the input feature map sequence 606, rather than the scale of the first-dimensional receptive field of one of the at least one second convolutional neural network operations that takes the input feature map sequence 606 as input, the multi-scale first-dimensional receptive field 802, 804, and 818 (not shown) has the finest scale. By fusing the first-dimensional importance 1010 and the input feature map sequence 606, information from the combination 810 of the multi-scale first-dimensional receptive fields 802, 804, and 818 (not shown) is fused.

[0066] like Fig.10 As shown, correspondingly, the multi-fused multi-first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 704 includes a multi-fused important and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 1006. The multi-fused important and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 1006 is configured to perform each third convolutional neural network operation using the fused important and finest scale first-dimension receptive field information 1012 to obtain the fused important and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information 1014. The corresponding fused important and finest first-dimension scale and single second-dimension scale first and second-dimension receptive field information 1014 of all third convolutional neural network operations is the information 608 from the multi-scale first and second-dimension receptive fields.

[0067] Accordingly, the multi-scale first and second dimensional receptive field information fusion module 604 includes an important and finest first dimension scale and an important second dimension scale first and second dimensional receptive field information fusion module 1008. The important and finest first dimension scale and the important second dimension scale first and second dimensional receptive field information fusion module 1008 is configured to fuse the information from the multi-scale first and second dimensional receptive field 608 to obtain important and finest first dimension scale and important second dimension scale first and second dimensional receptive field information 1016. The important first dimension scale and the important second dimension scale first and second dimensional receptive field information are also referred to as first and second dimensional importance in this application.

[0068] Figure 8 The corresponding combination of the receptive fields in the example is applicable to the present embodiment after necessary modifications, Figure 8 The corresponding combination of receptive fields in the above text corresponds to Figure 7 The multi-fusion multi-first dimension scale and single second dimension scale first and second dimension receptive field information acquisition module 704 and Figure 6 The corresponding steps performed by the multi-scale first and second dimensional receptive field information fusion module 604.

[0069] The first dimension importance and input feature map sequence fusion module 1004 is optional. In some embodiments, the fusion multi-scale first dimension receptive field information acquisition module 702 only includes the first dimension importance acquisition module 1002.

[0070] Fig.11 According to an embodiment of the present application Fig.10Schematic diagram of details of the first dimension importance acquisition module 1002 in FIG. The first dimension importance acquisition module 1002 is configured to perform at least one second convolutional neural network operation using the input feature map sequence 606 to obtain the first dimension importance 1010. In one embodiment, the input feature map sequence 606 is an input feature map sequence 1114. The size of the input feature map sequence 1114 is T x H x W, where T, H and W represent the time, height and width dimensions, respectively. At least one second convolutional neural network operation is a spatial pooling operation set 1102. The spatial pooling operation set 1102 includes multiple spatial pooling operations 1102a, 1102b and 1102c. The spatial pooling operations 1102a, 1102b and 1102c have different scales to capture spatial information of different resolutions. The spatial pooling operation 1102a uses a pooling window size of H x W to spatially pool the input feature map sequence 1114 into an output feature map sequence 1116a of the spatial pooling operation 1102a, wherein the size of the output feature map sequence 1116a of the spatial pooling operation 1102a is T x 1 x 1. The spatial pooling operation 1102b uses a pooling window size of (H / 2) x (W / 2) to spatially pool the input feature map sequence 1114 into an output feature map sequence 1116b of the spatial pooling operation 1102b, wherein the size of the output feature map sequence 1116b of the spatial pooling operation 1102b is T x 2 x2. The spatial pooling operation 1102c spatially pools the input feature map sequence 1114 into an output feature map sequence 1116c of the spatial pooling operation 1102c using a pooling window size of (H / 4) x (W / 4), wherein the output feature map sequence 1116c of the output feature map spatial pooling operation 1102c has a size of T x 4 x 4. Preferably, each spatial pooling operation 1102a, 1102b, or 1102c is a maximum pooling operation because the first-dimensional importance acquisition module 1002 attempts to capture the highest response to determine which region is more important for video classification. Alternatively, each spatial pooling operation 1102a, 1102b, or 1102c can be an average pooling operation.

[0071] Compared to the above embodiment in which at least one second convolutional neural network operation is a set of spatial pooling operations 1102, an alternative embodiment in which at least one second convolutional neural network operation is a set of spatial dilated convolution operations including multiple spatial dilated convolution operations is as follows. The above embodiment achieves multi-scale spatial receptive fields through corresponding different window sizes of spatial pooling operations. The alternative embodiment achieves multi-scale spatial receptive fields through corresponding different dilation rates of spatial dilated convolution operations.

[0072] After executing the spatial pooling operation set 1102, the first dimension importance acquisition module 1002 further performs a rescaling operation set 1104, a concatenating operation 1106, and then performs a channel size reducing (CSR) operation 1108 to fuse the output feature map sequence 1116 of the spatial pooling operation set 1102. The output feature map sequence 1116a, 1116b or 1116c of each corresponding spatial pooling operation 1102a, 1102b or 1102c is spatially subsampled. The rescaling operation set 1104 includes a plurality of rescaling operations 1104a, 1104b and 1104c. Each rescaling operation 1104a, 1104b or 1104c rescales the corresponding output feature map sequence 1116a, 1116b or 1116c of the corresponding spatial pooling operation 1102a, 1102b or 1102c back to the corresponding output feature map sequence 1118a, 1118b or 1118c of each rescaling operation 1104a, 1104b or 1104c. The size of the output feature map sequence 1118a, 1118b or 1118c of each corresponding rescaling operation 1104a, 1104b or 1104c is T x H x W. Each rescaling operation 1104a, 1104b or 1104c can be an interpolation operation, such as a bilinear interpolation operation. Alternatively, each rescaling operation 1104a, 1104b or 1104c can be a deconvolution operation. The concatenation operation 1106 concatenates the corresponding output feature map sequences 1118a, 1118b, and 1118c of the rescaling operation set 1104 along the channel axis, so that the output feature map sequence 1120 of the concatenation operation 1106 is of size T x H x W x 3. The CSR operation 1108 is applied to the output feature map sequence 1120 of the concatenation operation 1106 to obtain an output feature map sequence 1122 of the CSR operation 1108 of size T x H x W x 1. The CSR operation 1208 can be a 1 x 1 x 1 convolution operation. Alternatively, the CSR operation 1208 can be an element-wise summation operation. For an alternative embodiment in which at least one second convolutional neural network operation is a spatially expanded convolution operation set, the rescaling operation set 1104 may not be required.

[0073] After performing the CSR operation 1108, the first dimension importance acquisition module 1002 further uses the output feature map sequence 1122 of the CSR operation 1108 to perform a spatial attention map sequence obtaining operation 1110 to obtain a spatial attention map sequence 1124. The spatial attention map sequence obtaining operation 1110 can be a softmax function. Alternatively, the spatial attention map sequence obtaining operation 1110 can be a sigmoid function. Then, the first dimension importance acquisition module 1002 further performs an element-by-element multiplication operation 1112 on the input feature map sequence 1114 and the spatial attention map sequence 1124 to obtain spatial importance 1126. In this embodiment, the first dimension importance 1010 is the spatial importance 1126.

[0074] Compared to the above embodiment of performing the spatial attention map sequence acquisition operation 1110 to acquire the spatial importance 1126, an alternative embodiment of acquiring the spatial importance 1126 without the spatial attention map sequence acquisition operation 1110 is as follows. The output feature map sequence 1122 of the CSR operation 1108 is directly used as the spatial importance 1126.

[0075] Fig.12 This is a diagram showing an embodiment of the present application. Fig.10 A block diagram showing details of the multiple fusion important and finest first dimension scale and single second dimension scale first and second dimensional receptive field information acquisition module 1006 and the important and finest first dimension scale and important second dimension scale first and second dimensional receptive field information fusion module 1008.

[0076] The multiple fused important and finest first-dimensional scale and single second-dimensional scale first and second-dimensional receptive field information acquisition module 1006 is configured to use the fused important and finest scale first-dimensional receptive field information 1012 to perform each operation in the third convolutional neural network operation to obtain the corresponding fused important and finest first-dimensional scale and single second-dimensional scale first and second-dimensional receptive field information 1014. In one embodiment, the fused important and finest scale first-dimensional receptive field information 1012 is the fused important and finest scale spatial receptive field information 1210. The size of the fused important and finest scale spatial receptive field information 1210 is T x H x W, where T, H and W represent the time, height and width dimensions respectively. The third convolutional neural network operation is a temporal pooling operation set 1202. The temporal pooling operation set 1202 includes a plurality of temporal pooling operations 1202a, 1202b and 1202c. The temporal pooling operations 1202a, 1202b and 1202c have different scales to capture temporal information of different resolutions. The temporal pooling operation 1202a uses a pooling window of size T to temporally pool the fused important and finest scale spatial receptive field information 1210 into the output feature atlas 1212a of the temporal pooling operation 1202a, wherein the size of the output feature atlas 1212a of the temporal pooling operation 1202a is 1 x H x W. The temporal pooling operation 1202b uses a pooling window of size T / 2 to temporally pool the fused important and finest scale spatial receptive field information 1210 into the output feature atlas 1212b of the temporal pooling operation 1202b, wherein the size of the output feature atlas 1212b of the temporal pooling operation 1202b is 2 x H x W. The temporal pooling operation 1202c uses a pooling window of size T / 4 to temporally pool the fused important and finest scale spatial receptive field information 1210 into the output feature atlas 1212c of the temporal pooling operation 1202c, wherein the size of the output feature atlas 1212c of the temporal pooling operation 1202c is 4 x H x W. Each temporal pooling operation 1202a, 1202b, or 1202c is an average pooling operation. Alternatively, each temporal pooling operation 1202a, 1202b, or 1202c may be a maximum pooling operation. In this embodiment, the corresponding fused important and finest first-dimensional scale and single second-dimensional scale first and second-dimensional receptive field information 1014 is each of the output feature atlas 1212a of the temporal pooling operation 1202a, the output feature atlas 1212b of the temporal pooling operation 1202b, and the output feature atlas 1212c of the temporal pooling operation 1202c.

[0077] Compared to the above embodiment in which the third convolutional neural network operation is a temporal pooling operation set 1202, an alternative embodiment in which the third convolutional neural network operation is a temporal dilation convolution operation set including multiple temporal dilation convolution operations is as follows. The above embodiment implements a multi-scale temporal receptive field through corresponding different window sizes of the temporal pooling operation. The alternative embodiment implements a multi-scale temporal receptive field through corresponding different dilation rates of the temporal dilation convolution operation.

[0078] The important and finest first-dimension scale and the important second-dimension scale first and second-dimension receptive field information fusion module 1008 is configured to fuse the information 608 from the multi-scale first and second-dimension receptive fields to obtain the important and finest first-dimension scale and the important second-dimension scale first and second-dimension receptive field information 1016. In this embodiment, the information 608 from the multi-scale first and second-dimension receptive fields is the output feature atlas 1212a of the time pooling operation 1202a, the output feature atlas 1212b of the time pooling operation 1202b, and the output feature atlas 1212c of the time pooling operation 1202c. The important and finest first-dimension scale and the important second-dimension scale first and second-dimension receptive field information fusion module 1008 performs the rescaling operation set 1204, the connection operation 1206, and then performs the CSR operation 1208 to fuse the output feature atlas 1212 of the time pooling operation set 1202. The output feature atlas 1212a, 1212b, or 1212c of each corresponding temporal pooling operation 1202a, 1202b, or 1202c is temporally subsampled. The rescaling operation set 1204 includes a plurality of rescaling operations 1204a, 1204b, and 1204c. Each rescaling operation 1204a, 1204b, or 1204c rescales the corresponding output feature atlas 1212a, 1212b, or 1212c of the corresponding temporal pooling operation 1202a, 1202b, or 1202c back to the corresponding output feature map sequence 1210a, 1210b, or 1210c of each rescaling operation 1204a, 1204b, or 1204c. The size of the output feature map sequence 1210a, 1210b, or 1210c of each corresponding rescaling operation 1204a, 1204b, or 1204c is T x H x W. Each rescaling operation 1204a, 1204b, or 1204c may be an interpolation operation, such as a bilinear interpolation operation. Alternatively, each rescaling operation 1204a, 1204b, or 1204c may be a deconvolution operation. The connection operation 1206 connects the corresponding output feature map sequences 1210a, 1210b, and 1210c of the rescaling operation set 1204 along the channel axis, so that the size of the output feature map sequence 1216 of the connection operation 1206 is T x H x W x 3. The CSR operation 1208 is applied to the output feature map sequence 1216 of the connection operation 1206 to obtain the important and finest spatial scale and important time scale spatiotemporal receptive field information 1218 of size T x H x W x 1. The CSR operation 1208 may be a 1 x 1 x 1 convolution operation. Alternatively, the CSR operation 1208 may be an element-wise summation operation. In this embodiment, the important and finest first-dimensional scale and the important second-dimensional scale first and second-dimensional receptive field information 1016 are the important and finest spatial scale and the important time scale spatiotemporal receptive field information 1218.For alternative embodiments in which the third convolutional neural network operation is a set of time-dilated convolution operations, the rescaling operation set 1204 may not be required.

[0079] Compared with the above-mentioned embodiment in which the output feature map sequence of the CSR operation 1208 is directly used as the important and finest spatial scale and important time scale spatiotemporal receptive field information 1218, an alternative embodiment of obtaining the important and finest spatial scale and important time scale spatiotemporal receptive field information 1218 through the temporal attention map sequence acquisition operation is as follows. The important and finest first dimension scale and the important second dimension scale first and second dimensional receptive field information fusion module 1008 further uses the output feature map sequence of the CSR operation 1208 to perform a temporal attention map sequence acquisition operation to obtain a temporal attention map sequence. The temporal attention map sequence acquisition operation can be a softmax function. Alternatively, the temporal attention map sequence acquisition operation can be a sigmoid function. Then, the important and finest first dimension scale and the important second dimension scale first and second dimensional receptive field information fusion module 1008 further performs an element-by-element multiplication operation on the fused important and finest scale spatial receptive field information 1210 and the temporal attention map sequence 524 to obtain the important and finest spatial scale and important time scale spatiotemporal receptive field information 1218.

[0080] The term “attention map sequence mechanism” used in this application refers to the temporal attention map sequence acquisition operation and Fig.11 The spatial attention map sequence in obtains one or both of operations 1110.

[0081] And refer to Fig.11 and Fig.12 Compared to the above embodiment described where the first dimension is space and the second dimension is time, an alternative embodiment where the first dimension is space and the second dimension is time is as follows. Fig.10 The details of the first dimension importance acquisition module 1002 in the embodiment are modified as necessary and refer to Fig.12 Same as described. Fig.10 The multiple fusion important and most refined first dimension scale and single second dimension scale first and second dimensional receptive field information acquisition module 1006 and the important and most refined first dimension scale and important second dimension scale first and second dimensional receptive field information fusion module 1008 in the embodiment of the present invention are modified as necessary and refer to Fig.11 Same as described.

[0082] Fig.136 is a block diagram showing a module 1300 for acquiring multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information according to an embodiment of the present application. The module 1300 for acquiring multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information is configured to acquire fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information 1304 using an input feature map sequence 606. The module 1300 for acquiring multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information includes Figure 6 The fusion multi-scale first and second dimensional receptive field information acquisition module 600 in the embodiment. Fig.13 In the multi-scale first and second dimensional receptive field information acquisition module 602 in the fusion multi-scale first and second dimensional receptive field information acquisition module 600, the multi-scale first and second dimensional receptive field information acquisition module 602 includes Figure 7 The fused multi-scale first dimension receptive field information acquisition module 702 and the multi-fused multi-first dimension scale and single second dimension scale first and second dimension receptive field information acquisition module 704 in the embodiment. The fused multi-scale first and second dimension receptive field information and the multi-scale first dimension receptive field information acquisition module 1300 also includes a fused multi-scale first and second dimension receptive field information and a fused multi-scale first dimension receptive field information fusion module 1302. The fused multi-scale first and second dimension receptive field information and the fused multi-scale first dimension receptive field information fusion module 1302 is configured to fuse the fused multi-scale first and second dimension receptive field information 610 and the fused multi-scale first dimension receptive field information 706 into the fused multi-scale first and second dimension receptive field information and the multi-scale first dimension receptive field information 1304. In one embodiment, the fusion performed by the fused multi-scale first and second dimension receptive field information and the fused multi-scale first dimension receptive field information fusion module 1302 is an element-wise sum operation. In some of the above embodiments, the fused multi-scale first and second dimension receptive field information 610 includes first and second dimensional importance. The fused multi-scale first dimension receptive field information 706 includes the first dimension importance 1010 and the input feature map sequence 606. By fusing the fused multi-scale first and second dimension receptive field information 610 and the fused multi-scale first dimension receptive field information 706, the first and second dimension importance and the first dimension importance 1010 are added to the input feature map sequence 606 as residual information.

[0083] As mentioned above, the related art has the disadvantage that it cannot treat many irrelevant or unimportant information in each frame of the video clip as irrelevant or unimportant. Figures 14 to 17 For an example, see Figures 9 to 12 Effects of the described embodiments. In these examples, the first dimension is space and the second dimension is time.

[0084] like Fig.13 As shown, assuming that the fusion time importance and input feature map sequence acquisition module are Fig.13The fusion multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information acquisition module 1300 in the embodiment is similar, but the difference from the fusion multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information acquisition module 1300 is that the fusion time importance and input feature map sequence acquisition module does not include the fusion multi-scale first dimensional receptive field information acquisition module 702. That is, the multi-fusion multi-first dimensional scale and single second dimensional scale first and second dimensional receptive field information acquisition module 704 and the fusion multi-scale first and second dimensional receptive field information and fusion multi-scale first dimensional receptive field information fusion module 1302 directly or with appropriate modifications use the input feature map sequence 606 instead of the fusion multi-scale first dimensional receptive field information 706. The fusion time importance and input feature map sequence acquisition module is configured to use the input feature map sequence 606 to obtain the fusion time importance and input feature map sequence.

[0085] Fig.14 It is a schematic diagram showing a context with at least one more suitable time scale obtained by a module for obtaining a sequence of fused time importance and input feature maps according to an embodiment of the present application. Fig.14 is a simplified example for illustrating a context having at least one more appropriate time scale. In this example, temporal pooling operations 1202a and 1202b are shown, and for simplicity, temporal pooling operation 1202c is not shown. The input feature map sequence 606 is represented by an input feature map sequence 1402, which reflects a video clip with an athlete throwing a baseball. By performing a temporal pooling operation 1202a with a pooling window size of T on the input feature map sequence 1402, the output feature map 1404 of the temporal pooling operation 1202a reflects the motion profile of the entire input feature map sequence 1402, which is the context having the coarsest first time scale. By performing a time pooling operation 1202b with a pooling window size of T / 2 on the input feature map sequence 1402, each output feature map 1406a or 1406b in the output feature map set 1406 of the time pooling operation 1202b reflects the motion profile of the corresponding half of the input feature map sequence 1402, and the motion profile is a context with a second time scale. The second time scale is not as coarse as the first time scale.

[0086] When the computer vision CNN decides to classify a video clip with a player throwing a baseball using the fused temporal importance and input feature map sequence, the computer vision CNN can learn the following from the fused temporal importance and input feature map sequence. The fused temporal importance and input feature map sequence has the following fused parts: input feature map 1402a, output feature map 1406a of temporal pooling operation 1202b, and output feature map 1404 of temporal pooling operation 1202a. From the input feature map 1402a, the computer vision CNN can learn that there is a throwing action, but may not be able to learn whether the ball or the Frisbee is thrown. From the output feature map 1406a of the temporal pooling operation 1202b, the computer vision CNN can learn that someone and the ball are moving. From the output feature map 1404 of the temporal pooling operation 1202a, the computer vision CNN can learn that someone is moving. In this case, the output feature map 1406a of the temporal pooling operation 1202b may be a context having at least one more appropriate temporal scale, which is simultaneously considered with the input feature map 1402a for the computer vision CNN to learn that the person is throwing the ball. The output feature map 1404 of the temporal pooling operation 1202a may be too coarse to help identify the baseball throwing action. By simultaneously considering the context having at least one more appropriate temporal scale and the input feature map 1402a, less information is required in the fused temporal importance and input feature map sequence to identify the baseball throwing action in the video clip, and the remaining information that is not required for identifying the baseball throwing action in the fused temporal importance and input feature map sequence is considered irrelevant or unimportant.

[0087] The fused multi-scale first-dimensional receptive field information acquisition module 702 added in the fused multi-scale first and second-dimensional receptive field information and multi-scale first-dimensional receptive field information acquisition module 1300 has the following effects.

[0088] Fig.15 is a schematic diagram showing the background suppression effect of the fusion multi-scale first-dimensional receptive field information acquisition module 702 according to an embodiment of the present application. Fig.11 By describing the set of spatial pooling operations 1102 to capture spatial importance 1126, the computer vision CNN is forced to focus on the important spatial regions in the video clip, which are reflected in the fusion of the first dimension receptive field information 1012 of the important and finest scales. Fig.15In the example of , the irrelevant background in the video clip reflected in the input feature map sequence 1402 is represented by the shaded area in the input feature map sequence 1402. Since the computer vision CNN is forced to focus on the baseball and the players reflected in the fused important and finest scale first dimension receptive field information 1012, the fused important and finest scale first dimension receptive field information 1012 is represented by the fused important and finest scale first dimension receptive field information 1502, in which the irrelevant background is removed.

[0089] For the fusion of the important and finest scale first dimension receptive field information 1012, the spatial importance 1126 is added to the input feature map sequence 606 as residual information, providing the computer vision CNN with more contexts with different spatial scales. Therefore, the following effect is achieved: when making a decision to classify a video clip using the input feature map sequence 606, the computer vision CNN has a better chance to simultaneously consider a more appropriate context, including a context with at least one more appropriate spatial scale.

[0090] Fig.14 The temporal pooling operations 1202a and 1202b shown in FIG. Figure 1A and Figure 1B The convolutional neural network operation with spatial local-to-local constraints described in Fig.14 The described example uses a fusion time importance and input feature map sequence acquisition module that does not include the fusion multi-scale first dimension receptive field information acquisition module 702, so Fig.14 Each of the temporal pooling operations 1402a and 1402b in can only see the corresponding collocated local spatial region in the input feature map sequence 1402. In this way, the module for fusing temporal importance and input feature map sequence acquisition can obtain context with at least one more suitable temporal scale, but may not be able to obtain context with at least one more suitable spatiotemporal scale.

[0091] Fig.16 is a diagram showing a method according to an embodiment of the present application. Fig. 9 Schematic diagram of the context with global and finest spatial scales and important time scales obtained by the fusion multi-scale first and second dimensional receptive field information acquisition module 600 in FIG. Fig.16 The examples in follow Figure 1A and Figure 1B As previously seen in Figure 1A and Figure 1B As described, the time scale of the context provided by operation 106b of the first convolutional neural network operation set is more appropriate than the time scale of the context provided by operation 106a of the first convolutional neural network operation set. Fig.16 Only one convolutional neural network operation 1606 is shown, and the temporal pooling window size of the convolutional neural network operation 1606 is the same as the temporal pooling window size of the operation 106b of the first convolutional neural network operation set. Fig. 9 An example of one operation of the third convolutional neural network operation performed by the multiple fusion multiple first-dimensional scales and single second-dimensional scale first and second-dimensional receptive field information acquisition module 704 in the fusion multi-scale first and second-dimensional receptive field information acquisition module 600. Compared with the operation 106b of the first convolutional neural network operation set performed on the input feature map sequence 102, the convolutional neural network operation 1606 is performed on the fusion global and finest scale first-dimensional receptive field feature map sequence 1602. Fig.16 The fused global and finest scale first dimension receptive field feature map sequence 1602 shown in Fig. 9 912. Thus, even though the convolutional neural network operation 1606 has a spatial local-to-local constraint, the spatial portion of the spatiotemporal receptive field of the convolutional neural network operation 1606 has global and finest scales, rather than a single original scale.

[0092] Take the local spatial region 1604b on which the convolutional neural network operation 1606 is performed due to the spatial local-to-local constraint as an example. Because the local spatial region 1604b is in one of the fused global and finest scale first dimensional receptive field feature maps 1602b in the fused global and finest scale first dimensional receptive field feature map sequence 1602, when the convolutional neural network operation 1606 is performed on the local spatial region 1604b, the convolutional neural network operation 1606 has a spatial receptive field 1608 at the finest scale (i.e., 1 x 1) and a spatial receptive field 1610 at the global scale (i.e., H x W). In this way, even though the convolutional neural network operation 1606 has the spatial local-to-local constraint, the convolutional neural network operation 1606 knows the entire flight trajectory 1612 of the baseball 104 in the video frame corresponding to the fused global and finest scale first dimensional receptive field feature maps 1602a, 1602b, 1602c, 1602d at the global scale. Therefore, Figure 1B Compared to operation 106b of the first set of convolutional neural network operations in , convolutional neural network operation 1606 provides a context with a more appropriate spatiotemporal scale. Fig.14 Examples and references described Figure 1A and Figure 1B Similar to the example described in Fig.14 Compared to the examples described, see Fig.16The described example has the following advantages. Due to the knowledge of the flight trajectory 1612, the context provided by the convolutional neural network operation 1606 has a more appropriate spatiotemporal scale than that provided by the reference Fig.14 The context described is enhanced with at least one more appropriate time scale. Fig. 9 In the fused multi-scale first and second dimensional receptive field information and the multi-scale first dimensional receptive field information 1304 acquired by the fused multi-scale first and second dimensional receptive field information acquisition module 600, the information required to identify the baseball throwing action is less than the information required to identify the baseball throwing action in the fused time importance and input feature map sequence. Fig. 9 In the fused multi-scale first and second dimensional receptive field information and the multi-scale first dimensional receptive field information 1304 acquired by the fused multi-scale first and second dimensional receptive field information acquisition module 600, the remaining information not required for identifying the baseball throwing action is considered to be irrelevant or unimportant.

[0093] Fig.17 is a diagram showing a method according to an embodiment of the present application. Fig.10 Schematic diagram of context with important spatial scales and important temporal scales acquired by the fused multi-scale first and second dimensional receptive field information acquisition module 600 in FIG. Because the scale of the spatial portion of the spatiotemporal receptive field of the convolutional neural network operation 1606 that allows the entire flight trajectory 1612 of the baseball 104 to be known is global, and the baseball 104 flies from the center of the video frame corresponding to the fused global and finest scale first dimensional receptive field feature map 1602a to the upper left corner of the video frame corresponding to the fused global and finest scale first dimensional receptive field feature map 1602d, the flight trajectory 1612 of the baseball 104 may not be clearly seen. Fig.16 Compared to the convolutional neural network operation 1606 performed on the fused global and finest scale first dimension receptive field feature map sequence 1602, Fig.17 The convolutional neural network operation 1706 in FIG. 1 is performed on the fusion of the first dimension receptive field feature map sequence 1702 of the important and finest scales. The convolutional neural network operation 1706 is performed by Fig.10 An example of one operation in the third convolutional neural network operation performed by the multiple fusion multiple first-dimensional scales and single second-dimensional scale first and second-dimensional receptive field information acquisition module 704 in the fusion multi-scale first and second-dimensional receptive field information acquisition module 600. Fig.17 The fused important and finest scale first dimension receptive field feature map sequence 1702 shown in Fig.101012 in the example of fusion of important and finest scale first dimension receptive field information. Thus, even though the convolutional neural network operation 1606 has a spatial local-to-local constraint, the spatial portion of the spatiotemporal receptive field of the convolutional neural network operation 1706 has important and finest scales instead of a single original scale. The important scale may have at least one more appropriate scale than the global scale.

[0094] Take the local spatial region 1704b on which the convolutional neural network operation 1706 is performed due to the spatial local-to-local constraint as an example. Because the local spatial region 1704b is in a fused important and finest scale first-dimensional receptive field feature map 1702b in the fused important and finest scale first-dimensional receptive field feature map sequence 1702, when the convolutional neural network operation 1706 is performed on the local spatial region 1704b, the convolutional neural network operation 1706 has a spatial receptive field 1708 of the finest scale (i.e., 1 x 1) and spatial receptive fields 1710, 1714, and 1716 of the important scales (i.e., H x W, (H / 2)x(H / 2), and (H / 4)x(W / 4)). Thus, even though the convolutional neural network operation 1706 has a spatial local-to-local constraint, the convolutional neural network operation 1706 knows the entire flight trajectory 1712 of the baseball 104 in the video frames corresponding to the fused important and finest scale first dimension receptive field feature maps 1702a, 1702b, 1702c, and 1702d at the (H / 2)x(H / 2) scale. Figure 1B Compared to operation 106b of the first set of convolutional neural network operations in FIG. 1706, convolutional neural network operation 1706 provides a context with a more appropriate spatiotemporal scale. Fig.16 For comparison with the examples described, see Fig.17 The described example has the following advantages. The flight trajectory 1712 of baseball 104 is more clearly seen due to the knowledge of the flight trajectory 1712 of baseball 104 at the (H / 2)x(H / 2) scale. Therefore, the context provided by convolutional neural network operation 1706 is at a more appropriate spatiotemporal scale than the context provided by convolutional neural network operation 1606. Therefore, using Fig.10 The fused multi-scale first and second dimensional receptive field information and the multi-scale first dimensional receptive field information 1304 obtained by the fused multi-scale first and second dimensional receptive field information acquisition module 600 in the video clip have more information than that required by using Fig. 9 In the fused multi-scale first and second dimensional receptive field information and the multi-scale first dimensional receptive field information 1304 acquired by the fused multi-scale first and second dimensional receptive field information acquisition module 600, less information is required to recognize the baseball throwing action in the video clip. Fig.10In the fused multi-scale first and second dimensional receptive field information and the multi-scale first dimensional receptive field information 1304 acquired by the fused multi-scale first and second dimensional receptive field information acquisition module 600, the remaining information not required for identifying the baseball throwing action in the video clip is deemed irrelevant or unimportant.

[0095] In the description of this application, the term "importance" or "importance" refers to a scale from coarse to fine, wherein at least one scale provides a more appropriate scale for a computer vision CNN to treat irrelevant or unimportant information in each frame of a video clip as irrelevant or unimportant. An example of a coarse to fine temporal scale is the above referenced Fig.14 The first time scale and the second time scale are provided. Examples of spatial scales from coarse to fine are described above. Fig.17 Provided are the scales HxW, (H / 2)x(H / 2) and (H / 4)x(W / 4).

[0096] Fig.18 This is a diagram showing an embodiment of the present application. Figure 6 The multi-scale first and second dimensional receptive field information acquisition module 602 includes Fig.18 The multi-scale first and second dimensional receptive field information acquisition module 602 includes a multi-single first dimensional scale and a single second dimensional scale first and second dimensional receptive field information acquisition module 1802.

[0097] The multiple single first-dimension scale and single second-dimension scale first and second-dimension receptive field information acquisition module 1802 is configured to perform each of the multiple fourth convolutional neural network operations using the input feature map sequence 606 to obtain corresponding single first-dimension scale and single second-dimension scale first and second-dimension receptive field information 1804. Each first convolutional neural network operation set is a corresponding multiple operation set of the fourth convolutional neural network operation that provides a corresponding multi-scale second-dimension receptive field, wherein each multiple operation set provides all multi-scale first-dimension receptive fields and a corresponding multi-scale second-dimension receptive field. The corresponding single first-dimension scale and single second-dimension scale first and second-dimension receptive field information 1804 of all fourth convolutional neural network operations is information from the multi-scale first and second-dimension receptive fields 608. In one embodiment, each fourth convolutional neural network operation is a corresponding 3D pooling operation.

[0098] Fig.19 is a diagram showing a combination of receptive fields according to an embodiment of the present application, the combination of receptive fields corresponding to Fig.18 Module 1802 in the multi-scale first and second dimensional receptive field information acquisition module 602 and Figure 6The steps performed by module 604 in the fusion multi-scale first and second dimensional receptive field information acquisition module 600. Figure 8 In the example of , for multiple single first-dimension scales and single second-dimension scales first and second-dimension receptive field information acquisition modules 1802, each fourth convolutional neural network operation is performed. Each fourth convolutional neural network operation has a corresponding single first-dimension scale and single second-dimension scale first and second-dimension receptive field 1910, 1912, 1914 or 1916. The single first-dimension scale and single second-dimension scale first and second-dimension receptive field 1910 has a single first-dimension scale portion 1902 and a single second-dimension scale portion 1906. The single first-dimension scale and single second-dimension scale first and second-dimension receptive field 1912 has a single first-dimension scale portion 1904 and a single second-dimension scale portion 1906. The single first-dimension scale and single second-dimension scale first and second-dimension receptive field 1914 has a single first-dimension scale portion 1902 and a single second-dimension scale portion 1908. The single first-dimension scale and single second-dimension scale first and second-dimension receptive field 1916 has a single first-dimension scale portion 1904 and a single second-dimension scale portion 1908. Each first convolutional neural network operation set is a corresponding multiple operation set of a fourth convolutional neural network operation. Each multiple operation set has a corresponding combination 1918 or 1920 of each of all multi-scale first-dimensional receptive fields 1902 and 1904 and multi-scale second-dimensional receptive fields 1906 and 1908. The corresponding single first-dimensional scale and single second-dimensional scale first and second-dimensional receptive field information 1910, 1912, 1914, and 1916 of all fourth convolutional neural network operations is information 608 from multi-scale first and second-dimensional receptive fields.

[0099] For the multi-scale first and second dimensional receptive field information fusion module 604, information 608 from the combination of the multi-scale first and second dimensional receptive fields is fused to obtain fused multi-scale first and second dimensional receptive field information 610. That is, information from the single first dimensional scale and the single second dimensional scale first and second dimensional receptive fields 1910, 1912, 1914, and 1916 is fused. That is, information from the combination 1918 of all the multi-scale first dimensional receptive fields 1902 and 1904 and one 1906 of the multi-scale second dimensional receptive fields 1906 and 1908 and the combination 1922 of all the multi-scale first dimensional receptive fields 1902 and 1904 and the other 1908 of the multi-scale second dimensional receptive fields 1906 and 1908 is fused.

[0100] See also Figures 9 to 17 The described embodiments and effects are applicable with appropriate modifications to the Figure 18 to Figure 19 Described embodiments.

[0101] Fig. 20is a schematic diagram showing a separable 3DResnet-50 network 2000 having a plurality of modules G1 to G2 and H1 to H3 for acquiring multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information according to an embodiment of the present application. In one embodiment, each of the modules G1 to G2 and H1 to H3 for acquiring multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information is of a type Fig.13 The fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information acquisition module 1300 in the embodiment. The separable 3D Resnet-50 network 2000 is configured to classify the video clip 2002 to obtain a classification output 2004. The separable 3D Resnet-50 network 2000 includes a plurality of stages A to F connected in series to each other in sequence. Stages B to E are similar. Each of stages B to E is referred to as stage X to describe common features, and is referred to as corresponding stage B, ... or E to describe specific features. Stage A includes a spatial convolution layer A1 and a temporal convolution layer A2 connected in sequence to each other. Stage X includes a plurality of residual blocks connected in sequence to each other. The type of each residual block in a portion of the residual blocks before the last residual block is a 3D residual block 2100, which will be referred to in detail. Fig.21 The 3D residual block 2100 is described. The last residual block type is the 3D residual block 2200, which will be referred to in Fig. 22 The 3D residual block 2200 is described. Stages B, C, D and E have two, three, five and two residual blocks of the type of 3D residual block 2100, respectively. Stages C and D are more similar. For stage C, a module G1 for obtaining multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information is inserted after residual block C1, and a corresponding module G2 for obtaining multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information is inserted after other residual blocks of stage C after module G1. For stage D, a module H1 for obtaining multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information is inserted after residual block D1, and a corresponding module H2 or H3 for obtaining multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information is inserted after other residual blocks of stage D after module H1. Stage F includes a global average pooling layer F1 and a fully connected layer F2 which are sequentially connected to each other.

[0102] Compared to the above embodiment of inserting the fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information acquisition modules G1 to G2 and H1 to H3 into stages C and D in the separable 3D Resnet-50 network 2000, alternative embodiments of inserting any number of fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information acquisition modules after any corresponding convolutional layer in any deep CNN architecture are within the scope of the present application. Each fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information acquisition module is a lightweight but general component for capturing the importance of the first and second dimensions in any deep CNN architecture.

[0103] Fig.212 is a block diagram showing a 3D residual block 2100 according to an embodiment of the present application. The 3D residual block 2100 is configured to use an input feature map sequence 2112 to perform spatial convolution and temporal convolution through a plurality of separate and independent convolution layers 2102, 2104, 2106, and 2108 to obtain an output feature map sequence 2114 of the 3D residual block 2100. The 3D residual block 2100 includes convolution layers 2102, 2104, 2106, and 2108 and an element-by-element summation operation 2110. The convolution layers 2102, 2104, 2106, and 2108 are sequentially connected in series. The element-by-element summation operation 2110 sums the output feature map sequence of the convolution layer 2108 and the input feature map sequence 2112. The size of the input feature map sequence 2112 is T x H x W x C, where T, H, W, and C represent time, height, width, and number of channels. The kernel size of convolution layer 2102 is 1 x 1 x 1, and the stride is 1 x 1 x 1. The size of the output feature map sequence of convolution layer 2102 is T x H x W x C / 4, and can be modified by a nonlinear activation function such as ReLU. The kernel size of convolution layer 2104 is 1 x 3 x 3, and the stride is 1 x 1 x 1. The size of the output feature map sequence of convolution layer 2104 is T x H x W x C / 4, and can be modified by a nonlinear activation function such as ReLU. The kernel size of convolution layer 2106 is 3 x 1 x 1, and the stride is 1 x 1 x 1. The size of the output feature map sequence of convolution layer 2106 is T x H x W x C / 4, and can be modified by a nonlinear activation function such as ReLU. The kernel size of convolution layer 2108 is 1 x 1 x 1, and the stride is 1 x 1 x 1. The size of the output feature map sequence of the convolution layer 2108 is T x H x W x C. The output feature map sequence of the element-wise summation operation 2110 can be modified by a nonlinear activation function such as ReLU to obtain an output feature map sequence 2114 of the 3D residual block 2100. The size of the output feature map sequence 2114 of the 3D residual block 2100 is T x H x W x C.

[0104] Fig. 222 is a block diagram showing a 3D residual block 2200 according to an embodiment of the present application. The 3D residual block 2200 is configured to use an input feature map sequence 2214 to perform downscaled spatial convolution and temporal convolution through a plurality of separate and independent convolution layers 2202, 2204, 2206, 2208, and 2210 to obtain an output feature map sequence 2216 of the 3D residual block 2200. The 3D residual block 2200 includes convolution layers 2202, 2204, 2206, 2208, and 2210 and an element-by-element summation operation 2212. The convolution layers 2202, 2204, 2206, 2208, and 2210 are sequentially connected in series. The element-by-element summation operation 2212 sums the output feature map sequence of the convolution layer 2210 and the input feature map sequence 2214. The size of the input feature map sequence 2214 is T x H x W x C, where T, H, W, and C represent time, height, width, and number of channels. The kernel size of the convolution layer 2202 is 1 x 1 x 1, and the stride is 2 x 2 x 2. The size of the output feature map sequence of the convolution layer 2202 is T / 2 x H / 2 x W / 2 x C, and can be modified by a nonlinear activation function such as ReLU. The kernel size of the convolution layer 2204 is 1 x 1 x 1, and the stride is 1 x 1 x 1. The size of the output feature map sequence of the convolution layer 2204 is T x H x W x C / 4, and can be modified by a nonlinear activation function such as ReLU. The kernel size of the convolution layer 2206 is 1 x 3 x 3, and the stride is 1 x 2 x 2. The size of the output feature map sequence of the convolution layer 2206 is T x H / 2 x W / 2 x C / 4, and can be modified by a nonlinear activation function such as ReLU. The kernel size of the convolution layer 2208 is 3 x 1 x 1, and the stride is 2 x 1 x 1. The kernel size of the convolution layer 2210 is 1 x 1 x 1, and the stride is 1 x 1 x 1. The size of the output feature map sequence of the convolution layer 2210 is T / 2 x H / 2 x W / 2 x C. The output feature map sequence of the element-wise sum operation 2212 can be modified by a nonlinear activation function such as ReLU to obtain an output feature map sequence 2216 of the 3D residual block 2200. The size of the output feature map sequence 2216 of the 3D residual block 2200 is T / 2 x H / 2 x W / 2 x C.

[0105] It is understood by those of ordinary skill in the art that each unit, module, layer, block, algorithm and step of the system or computer-implemented method described and disclosed in the embodiments of the present application is implemented using hardware, firmware, software or a combination thereof. Whether the function is run in hardware, firmware or software depends on the application conditions and design requirements of the technical solution. Those of ordinary skill in the art can use different methods to implement the functions of each specific application, and such implementation should not exceed the scope of the present invention. It is understood by those of ordinary skill in the art that since the working processes of the above-mentioned systems, devices, modules and units are basically the same, he / she can refer to the working processes of the systems, devices, modules and units in the above-mentioned embodiments. For ease of description and simplification, these working processes will not be described in detail.

[0106] It is understood that the systems, devices, and computer implementation methods disclosed in the embodiments of the present application can be implemented in other ways. The above embodiments are exemplary only. The division of units or modules is based only on logical functions, and other divisions exist in the implementation. Units or modules may or may not be physical units or modules. Multiple units or modules can be combined or integrated into one physical unit or module. Any unit or module can also be divided into multiple physical units or modules. Certain features can also be omitted or skipped. On the other hand, the mutual coupling, direct coupling, or communication coupling shown or discussed is operated through some ports, devices, units or modules, whether indirectly or through electrical, mechanical or other types of forms of communication.

[0107] The units or modules used as separate components for explanation may or may not be physically separated. These units or modules are located in one place or distributed over multiple network units or modules. Depending on the purpose of the embodiment, some or all of the units or modules are used. In addition, each functional unit or module in each embodiment may be integrated in a physically independent processing unit or module, or in a processing unit or module having two or more units or modules.

[0108] If the software functional unit or module is used and sold as a product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution proposed in this application can be implemented basically or partially in the form of a software product. Alternatively, a part of the technical solution that is advantageous to conventional technology can be implemented in the form of a software product. The software product is stored in a computer-readable storage medium, including multiple commands for a processor module of a computing device (such as a personal computer, a mobile phone) to run all or part of the steps disclosed in the embodiment of the present invention. The storage medium includes a USB (Universal Serial Bus, Universal Serial Bus) disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a floppy disk or other types of media capable of storing program instructions.

[0109] While the present invention has been described in connection with what is considered to be the most practical and preferred embodiment, it is to be understood that the application is not limited to the disclosed embodiment, but is intended to cover various arrangements that may be made without departing from the scope of the broadest interpretation of the appended claims.

Claims

1. A computer-implemented method, characterized in that: include: Fusing information from corresponding combinations of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field to obtain information from multi-scale first and second-dimensional receptive fields corresponding to the multi-scale second-dimensional receptive field; wherein the multi-scale first-dimensional receptive field of the first dimension is one of space and time, and the multi-scale second-dimensional receptive field of the second dimension is the other of space and time; The all multi-scale first-dimensional receptive fields include a first multi-scale first-dimensional receptive field and a second multi-scale first-dimensional receptive field, wherein the first multi-scale first-dimensional receptive field has a global scale, and the second multi-scale first-dimensional receptive field has a finest scale; Each of the multi-scale second-dimensional receptive fields is provided by a corresponding first convolutional neural network operation set, the first convolutional neural network operation set is one of a temporal pooling operation, a spatial pooling operation, and a 3D pooling operation, and each operation in the first convolutional neural network operation set has a corresponding first-dimensional local-to-local constraint; and Fusing the information from the combination of the multi-scale first and second dimensional receptive fields to obtain fused multi-scale first and second dimensional receptive field information; Wherein, obtaining the information from the multi-scale first and second dimensional receptive fields includes obtaining fused multi-scale first dimensional receptive field information; The fused multi-scale first and second dimensional receptive field information is used to fuse with the fused multi-scale first dimensional receptive field information to obtain fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information, the fused multi-scale first and second dimensional receptive field information is obtained using the first and second dimensional importance, and the fused multi-scale first dimensional receptive field information is obtained using the first dimension importance.

2. The computer-implemented method of claim 1, wherein: The step of fusing information from a corresponding combination of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field comprises: Performing at least one second convolutional neural network operation using the input feature map sequence to obtain the fused multi-scale first-dimensional receptive field information; wherein at least one of the multi-scale first-dimensional receptive fields is provided by a corresponding at least one second convolutional neural network operation, wherein a scale of the at least one multi-scale first-dimensional receptive field is greater than a minimum scale of the multi-scale first-dimensional receptive field, wherein the at least one second convolutional neural network operation is a set of spatial pooling operations or a set of spatial dilated convolution operations including a plurality of spatial dilated convolution operations; and Performing each of a plurality of third convolutional neural network operations using the fused multi-scale first-dimensional receptive field information to obtain corresponding fused multi-first-dimensional scale and single second-dimensional scale first-dimensional receptive field information; wherein each of the first convolutional neural network operation sets provides a corresponding third convolutional neural network operation of a corresponding multi-scale second-dimensional receptive field; and The corresponding fused multiple first-dimensional scales and single second-dimensional scale first and second-dimensional receptive field information of all the third convolutional neural network operations is the information from the multi-scale first and second-dimensional receptive fields.

3. The computer-implemented method of claim 2, wherein: The first dimension importance is obtained through multiple second convolutional neural network operations.

4. The computer-implemented method of claim 3, wherein: The fused multi-scale first-dimensional receptive field information is obtained by fusing the first-dimensional importance with the input feature map sequence.

5. The computer-implemented method of claim 1, wherein: The step of fusing information from a corresponding combination of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field comprises: Performing each of a plurality of fourth convolutional neural network operations using the input feature map sequence to obtain corresponding single first-dimensional scale and single second-dimensional scale first-dimensional receptive field information, wherein the plurality of fourth convolutional neural network operations are 3D pooling operations; wherein each of the first convolutional neural network operation sets is a corresponding multiple operation set in the fourth convolutional neural network operation that provides a corresponding multi-scale second-dimensional receptive field, wherein each of the multiple operation sets provides all of the multi-scale first-dimensional receptive fields and a corresponding multi-scale two-dimensional receptive field; The corresponding single first-dimensional scale and single second-dimensional scale first and second-dimensional receptive field information of all the fourth convolutional neural network operations is the information from the multi-scale first and second-dimensional receptive fields.

6. The computer-implemented method of claim 1, wherein: The multi-scale first-dimensional receptive field and each of the multi-scale second-dimensional receptive fields are provided by corresponding pooling operation sets, and each of the pooling operation sets includes the corresponding first convolutional neural network operation set.

7. The computer-implemented method of claim 1, wherein: The first and second dimension importances are obtained using an attention graph sequence mechanism.

8. A system, characterized in that: include: a memory module configured to store program instructions; A processor module is configured to execute the program instructions, wherein the program instructions cause the processor module to perform at least one step, wherein the at least one step comprises: Fusing information from corresponding combinations of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field to obtain information from multi-scale first and second-dimensional receptive fields corresponding to the multi-scale second-dimensional receptive field; wherein the multi-scale first-dimensional receptive field of the first dimension is one of space and time, and the multi-scale second-dimensional receptive field of the second dimension is the other of space and time; The all multi-scale first-dimensional receptive fields include a first multi-scale first-dimensional receptive field and a second multi-scale first-dimensional receptive field, wherein the first multi-scale first-dimensional receptive field has a global scale, and the second multi-scale first-dimensional receptive field has a finest scale; Each of the multi-scale second-dimensional receptive fields is provided by a corresponding first convolutional neural network operation set, the first convolutional neural network operation set is one of a temporal pooling operation, a spatial pooling operation, and a 3D pooling operation, and each operation in the first convolutional neural network operation set has a corresponding first-dimensional local-to-local constraint; and Fusing the information from the combination of the multi-scale first and second dimensional receptive fields to obtain fused multi-scale first and second dimensional receptive field information; Wherein, obtaining the information from the multi-scale first and second dimensional receptive fields includes obtaining fused multi-scale first dimensional receptive field information; The fused multi-scale first and second dimensional receptive field information is used to fuse with the fused multi-scale first dimensional receptive field information to obtain fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information, the fused multi-scale first and second dimensional receptive field information is obtained using the first and second dimensional importance, and the fused multi-scale first dimensional receptive field information is obtained using the first dimension importance.

9. The system according to claim 8, characterized in that The step of fusing information from a corresponding combination of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field comprises: Performing at least one second convolutional neural network operation using the input feature map sequence to obtain the fused multi-scale first-dimensional receptive field information; wherein at least one of the multi-scale first-dimensional receptive fields is provided by a corresponding at least one second convolutional neural network operation, wherein a scale of the at least one multi-scale first-dimensional receptive field is greater than a minimum scale of the multi-scale first-dimensional receptive field, wherein the at least one second convolutional neural network operation is a set of spatial pooling operations or a set of spatial dilated convolution operations including a plurality of spatial dilated convolution operations; and Performing each of a plurality of third convolutional neural network operations using the fused multi-scale first-dimensional receptive field information to obtain corresponding fused multi-first-dimensional scale and single second-dimensional scale first-dimensional receptive field information; wherein each of the first convolutional neural network operation sets provides a corresponding third convolutional neural network operation of a corresponding multi-scale second-dimensional receptive field; and The corresponding fused multiple first-dimensional scales and single second-dimensional scale first and second-dimensional receptive field information of all the third convolutional neural network operations is the information from the multi-scale first and second-dimensional receptive fields.

10. The system according to claim 9, characterized in that The first dimension importance is obtained through multiple second convolutional neural network operations.

11. The system according to claim 10, characterized in that The fused multi-scale first-dimensional receptive field information is obtained by fusing the first-dimensional importance with the input feature map sequence.

12. The system according to claim 8, characterized in that The step of fusing information from a corresponding combination of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field comprises: Performing each of a plurality of fourth convolutional neural network operations using the input feature map sequence to obtain corresponding single first-dimensional scale and single second-dimensional scale first-dimensional receptive field information, wherein the plurality of fourth convolutional neural network operations are 3D pooling operations; wherein each of the first convolutional neural network operation sets is a corresponding multiple operation set in the fourth convolutional neural network operation that provides a corresponding multi-scale second-dimensional receptive field, wherein each of the multiple operation sets provides all of the multi-scale first-dimensional receptive fields and a corresponding multi-scale two-dimensional receptive field; The corresponding single first-dimensional scale and single second-dimensional scale first and second-dimensional receptive field information of all the fourth convolutional neural network operations is the information from the multi-scale first and second-dimensional receptive fields.

13. The system according to claim 8, characterized in that The multi-scale first-dimensional receptive field and each of the multi-scale second-dimensional receptive fields are provided by corresponding pooling operation sets, and each of the pooling operation sets includes the corresponding first convolutional neural network operation set.

14. The system according to claim 8, characterized in that The first and second dimension importances are obtained using an attention graph sequence mechanism.

15. A non-transitory computer-readable medium, characterized in that The non-transitory computer readable medium has program instructions stored thereon, and when the program instructions are executed by the processor module, the processor module is caused to perform the following steps: Fusing information from corresponding combinations of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field to obtain information from multi-scale first and second-dimensional receptive fields corresponding to the multi-scale second-dimensional receptive field; wherein the multi-scale first-dimensional receptive field of the first dimension is one of space and time, and the multi-scale second-dimensional receptive field of the second dimension is the other of space and time; The all multi-scale first-dimensional receptive fields include a first multi-scale first-dimensional receptive field and a second multi-scale first-dimensional receptive field, wherein the first multi-scale first-dimensional receptive field has a global scale, and the second multi-scale first-dimensional receptive field has a finest scale; Each of the multi-scale second-dimensional receptive fields is provided by a corresponding first convolutional neural network operation set, the first convolutional neural network operation set is one of a temporal pooling operation, a spatial pooling operation, and a 3D pooling operation, and each operation in the first convolutional neural network operation set has a corresponding first-dimensional local-to-local constraint; and Fusing the information from the combination of the multi-scale first and second dimensional receptive fields to obtain fused multi-scale first and second dimensional receptive field information; Wherein, obtaining the information from the multi-scale first and second dimensional receptive fields includes obtaining fused multi-scale first dimensional receptive field information; The fused multi-scale first and second dimensional receptive field information is used to fuse with the fused multi-scale first dimensional receptive field information to obtain fused multi-scale first and second dimensional receptive field information and multi-scale first dimensional receptive field information, the fused multi-scale first and second dimensional receptive field information is obtained using the first and second dimensional importance, and the fused multi-scale first dimensional receptive field information is obtained using the first dimension importance.

16. The non-transitory computer readable medium of claim 15, wherein: The step of fusing information from a corresponding combination of all multi-scale first-dimensional receptive fields and each multi-scale second-dimensional receptive field comprises: Performing at least one second convolutional neural network operation using the input feature map sequence to obtain the fused multi-scale first-dimensional receptive field information; wherein at least one of the multi-scale first-dimensional receptive fields is provided by a corresponding at least one second convolutional neural network operation, wherein a scale of the at least one multi-scale first-dimensional receptive field is greater than a minimum scale of the multi-scale first-dimensional receptive field, wherein the at least one second convolutional neural network operation is a set of spatial pooling operations or a set of spatial dilated convolution operations including a plurality of spatial dilated convolution operations; and Performing each of a plurality of third convolutional neural network operations using the fused multi-scale first-dimensional receptive field information to obtain corresponding fused multi-first-dimensional scale and single second-dimensional scale first-dimensional receptive field information; wherein each of the first convolutional neural network operation sets provides a corresponding third convolutional neural network operation of a corresponding multi-scale second-dimensional receptive field; and The corresponding fused multiple first-dimensional scales and single second-dimensional scale first and second-dimensional receptive field information of all the third convolutional neural network operations is the information from the multi-scale first and second-dimensional receptive fields.

17. The non-transitory computer readable medium of claim 16, wherein: The first dimension importance is obtained through multiple second convolutional neural network operations.

18. The non-transitory computer readable medium of claim 17, wherein: The fused multi-scale first-dimensional receptive field information is obtained by fusing the first-dimensional importance with the input feature map sequence.

Images