Video understanding neural network system and its use method

Abstract

This application relates to a Temporal Information Aggregation (TIA) neural network block to extract latent multi-scale temporal information. By applying TIA, information at different time scales can be efficiently extracted. The TIA block can be implemented as a single block and can therefore be plugged into any architecture. The extracted multi-scale temporal information contributes to the final output as a residual.

Description

Technical Field

[0001] The present application relates to neural network systems and methods, and more particularly, to video understanding neural network systems and methods of using the same. Background Art

[0002] Video is an indispensable part of today's big data. Driven by the progress of deep learning for images and speech, using deep learning techniques to understand video content has become one of the hottest topics. Compared with still images, there are additional multi-scale temporal patterns hidden in video data. Video understanding, such as video classification or other types of machine learning methods using artificial intelligence, aims to extract robust video feature representations and effectively utilize multi-scale spatiotemporal cues in videos to automatically classify video clips according to their semantic content, such as human actions or complex events.

[0003] A large amount of work has been conducted on this task. However, existing works are usually affected by expensive computational costs or memory overhead. For example, two-stream CNNs are computationally expensive and cannot consider the order of frames and cannot analyze temporal dynamics in videos; non-local neural networks are characterized by large memory overhead in matrix multiplication; and slow networks are computationally expensive.

[0004] Therefore, there is a need to find an efficient, lightweight, yet general component for capturing temporal dependencies in deep neural networks. Summary of the invention

[0005] One aspect of the present application relates to a neural network system (e.g., a 2D, 3D neural network system or a neural network system of other dimensions) implemented by one or more electronic devices. The neural network includes: a baseline neural network for processing multiple images constructed in a time series. The baseline neural network includes: a first baseline block for processing at least one spatial feature in a first feature map, thereby converting the first feature map related to the multiple images into a second feature map; and a target neural network block adjacent to the first baseline block for performing temporal processing on the second feature map, thereby converting the second feature map into a temporal feature map; and combining the temporal feature map with the second feature map to generate a third feature map.

[0006] According to one aspect of the present application, a method for analyzing multiple images (e.g., video clips) constructed in a time series using at least one neural network is proposed. The method includes: a first block of a baseline neural network processes at least one spatial feature in a first feature map, thereby converting the first feature map associated with multiple images into a second feature map; a target neural network block performs temporal processing on the second feature map, thereby converting the second feature map into a temporal feature map; and the target neural network block combines the temporal feature map with the second feature map to generate a third feature map.

[0007] According to another aspect of the present application, a non-temporary storage medium may include an instruction set for a neural network (e.g., a two-dimensional, three-dimensional neural network system or a neural network system of other dimensions) and an operation instruction for operating the neural network. When executing the instruction set, the processor of the electronic device may perform the following operations: the first block of the baseline neural network processes at least one spatial feature in the first feature map, thereby converting the first feature map related to multiple images into a second feature map; the target neural network block performs temporal processing on the second feature map, thereby converting the second feature map into a temporal feature map; and the target neural network block combines the temporal feature map with the second feature map to generate a third feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The content of the present application will be further described by exemplary embodiments. In the following detailed description, when combined with the accompanying drawings, the above and other aspects of the embodiments of the present application will become more apparent.

[0009] Figure 1 A schematic block diagram of an electronic device for operating a neural network system and a video classification method proposed in an embodiment of the present application is shown;

[0010] Figure 2 A schematic diagram of the structure of a baseline neural network for classifying and analyzing video clips proposed in an embodiment of the present application is shown;

[0011] Figure 3A -C is a schematic block diagram showing different neural network block structures of the baseline neural network proposed in the embodiment of the present application;

[0012] Figure 4 A schematic diagram of the structure of a modified neural network for classifying video clips proposed in an embodiment of the present application is shown;

[0013] Figure 5 A schematic diagram of a time information aggregation block proposed in an embodiment of the present application is shown;

[0014] Fig. 6A -C is a schematic diagram showing the time information that can be extracted by the time information aggregation block proposed in the embodiment of the present application;

[0015] Figure 7 A flowchart of a method for analyzing multiple images in time series constructed by at least one neural network proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0016] One aspect of the present application relates to a temporal information aggregation (TIA) module or block. By continuously temporally pooling a feature map with various pooling windows, the TIA block can effectively extract potential multi-scale temporal information from the feature map. The extracted multi-scale temporal information may then contribute to the final output of a neural network or a portion of a neural network as a residual. The TIA module / block can be inserted into any neural network architecture (such as a ResNet-50 network) for various computer vision tasks such as video search, moment search, and subtitles. Depending on the time scale required to understand the video clip, the pooling window can be the same or different from the corresponding time scale of the kernel of the convolutional layer in the baseline neural network, so that the temporal information extracted by TIA is different from the time scale of the baseline neural network.

[0017] The following description is intended to enable those skilled in the art to make and use the contents of the present application, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present application. Therefore, the contents of the present application are not limited to the illustrated embodiments, but should be given the broadest scope consistent with the claims.

[0018] It should also be understood that, although the terms "first", "second", etc. may be used to describe various elements herein, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first contact may be referred to as a second contact, and similarly, a second contact may also be referred to as a first contact without departing from the scope of the present application. Both the first contact and the second contact are contacts, but they are not the same contact.

[0019] The terms used herein are only used to describe specific exemplary embodiments and are not intended to be limiting. As used herein, the singular form "a" or "an" may be intended to include the plural form as well, unless the context clearly indicates otherwise. It will be further understood that the terms "include", "may include" and / or "comprise" when used in this specification specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0020] As used herein, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [the condition or event] is detected" may be interpreted as "upon determination" or "in response to determining" or "upon detecting [the condition or event]" or "in response to detecting [the condition or event]," depending on the context.

[0021] As used herein, programs, instructions, and data are stored in a predetermined data structure. For example, the data structure may include bytes of a first part, a second part, and a third part. The second part may include the content to which the data relates. For example, for an image stored in a storage medium, its content data may be the substantial content of the image. For an instruction, its content may be the substantial content of the command corresponding to the instruction. The third part of the data may be a pointer end, and the pointer head may point to the first part of the next data byte. The first part of the data may be a pointer head, where the pointer end may be connected to the third data part of another data byte.

[0022] These and other features of the present application, as well as the methods of operation and functions of the related structural elements, and the combination of parts and the economy of manufacture, may become more apparent after considering the following description with reference to the accompanying drawings, all of which form a part of this specification. However, it should be expressly understood that the drawings are for illustration and description purposes only and do not limit the scope of the disclosure of the present application. It should be understood that the drawings are not drawn to scale.

[0023] The flow chart used in this application illustrates, according to some embodiments in this application, the operation of system implementation.It should be clearly understood that the operation of flow chart may or may not be implemented in sequence.On the contrary, these operations can be implemented in the reverse order, or implemented simultaneously.In addition, one or more other operations can be added to the flow chart.One or more operations can be removed from the flow chart.

[0024] In addition, although the systems and methods in this application are mainly described with respect to electronic devices such as computers, workstations, and servers, it should also be understood that this is only an exemplary implementation of the systems and methods described in this application. It will be understood by those skilled in the art at the time of the filing date of this application that the systems and methods of this application may also be implemented in other electronic devices, such as smart phones, tablet computers, laptop computers, desktop computers, smart cameras, or other electronic devices with video / image sequence processing capabilities.

[0025] Figure 1A schematic block diagram of an electronic device for operating a neural network system and a video classification method according to an embodiment of the present application is shown. For example, the electronic device 100 may be one or more computers, workstations, servers, smart phones, other mobile devices, or other electronic devices.

[0026] The portable device 100 may include a processor 120 (e.g., a CPU and / or a GPU and / or a DSP), a memory controller 122, a memory 102, a peripheral device interface 118, a power system 162, and an I / O subsystem 106 connected to the peripheral device interface 118. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 may be implemented on a single chip, such as the chip 104. In other embodiments, they may be implemented on separate chips, respectively.

[0027] The power system 162 can provide power to various components in the device 100, such as the CPU 120, the memory controller 122, the memory 102, the peripheral device interface 118, the power system 162, and a number of I / O subsystems 106 connected to the peripheral device interface 118. The power system 162 can include a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components related to the generation, management, and distribution of power in a portable device.

[0028] I / O subsystem 106 may include external port 124, audio device 110, external port 124, display 156, and optical sensor 158. Peripherals interface 118 may receive / send electrical signals from / to their respective devices in I / O subsystem 106.

[0029] Audio device 110 , such as a speaker and / or a microphone, may provide an audio interface between a user and device 100 .

[0030] Display 156 may provide an input / output interface to a user. In some embodiments, display 156 may be a screen or a touch-sensitive screen of device 100. Peripheral device interface 118 may receive and / or send electrical signals from / to touch-sensitive display system 112. Touch-sensitive display system 112 may display visual output (e.g., an image or video stream) to a user and / or receive instructions from a user.

[0031] The optical sensor 158 may be one or more built-in cameras that include one or more lenses and a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 158 may receive light from the environment, projected through one or more lenses, and convert the light into data representing an image. In conjunction with the imaging module 143 (also referred to as a camera module), the optical sensor 158 may capture still images and / or video streams.

[0032] Images or video streams may also be input to the peripherals interface 118 through other input devices 160 , such as internal and / or external memory that stores images and videos.

[0033] The memory 102 may include a mass storage, a removable memory, a volatile read-write memory, a read-only memory (ROM), or the like, or any combination thereof. For example, the mass storage may include a magnetic disk, an optical disk, a solid-state drive, or the like. The removable memory may include a flash drive, a floppy disk, an optical disk, a memory card, a compressed disk, a magnetic tape, or the like. The volatile read-write memory may include a random access memory (RAM). The RAM may include a dynamic RAM (DRAM), a double rate synchronous dynamic RAM (DDR SDRAM), a static RAM (SRAM), a thyristor RAM (T-RAM), and a zero capacitance RAM (Z-RAM), or the like. The ROM may include a mask ROM (MROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a compact disk ROM (CD-ROM), and a digital versatile disk ROM, or the like. In some embodiments, the memory 102 may store one or more software components to perform the exemplary methods described in the present application. For example, the memory 102 may store a program for the processor to process image data stored in the memory 102 or received by the processor 120 from an I / O device (e.g., a built-in camera).

[0034] In some embodiments, the one or more software components may include an operating system 126 , a communication module (or instruction set) 128 , a graphics module (or instruction set) 132 , and an application (or instruction set) 136 .

[0035] The operating system 126 (e.g., ANDROID, iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) may include various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitating communication between various hardware and software components.

[0036] The communication module 128 can facilitate communication with other devices through one or more external ports 124, and can also include various software components for processing data received from the external ports 124. The external port 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) can be suitable for direct coupling to other devices or indirect coupling through a network (e.g., the Internet, wireless local area network, etc.). In some embodiments, the external port can be a multi-pin (e.g., 30-pin) synthesizer.

[0037] The graphics module 132 may include various known software components for rendering and displaying graphics on the display 156, including components for changing the visual effects (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" may include any object that can be displayed to a user, including but not limited to text, web pages, icons (e.g., user interface objects including soft keys), digital images, videos, animations, etc.

[0038] In some embodiments, the graphics module 132 may store data representing graphics to be used. Each graphic may optionally be assigned a corresponding code. The graphics module 132 may receive data from an application or an optical sensor 158, etc.

[0039] Applications 136 may include the following modules (or instruction sets), or subsets or supersets thereof: camera module 143 for still and / or video images, image management module 144, and other applications 146. Examples of other applications 146 stored in memory 102 may include neural network applications, such as a 3-D ResNet-50 network or any modification based thereon.

[0040] In conjunction with the display 156, one or more optical sensors 158, the graphics module 132, and the image management module 144, the camera module 143 may include executable instructions to capture still images or videos (including video streams) from the optical sensor 158 (e.g., a camera) and store them in the memory 102, modify characteristics of the still images or videos, and / or delete the still images or videos from the memory 102.

[0041] In conjunction with display 156, graphics module 132, and camera module 143, image management module 144 may include executable instructions to arrange, modify (e.g., edit) or otherwise manipulate, tag, delete, present (e.g., in a digital slide show or album), and store still and / or video images.

[0042] Each of the above modules and applications may correspond to a set of executable instructions for performing one or more of the above functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not need to be implemented as separate software programs, programs, or modules, so various subsets of these modules may be selectively combined or otherwise reconstructed in various embodiments. In some embodiments, memory 102 may store subsets of the above modules and data structures. In addition, memory 102 may selectively store additional modules and data structures not described above.

[0043] Furthermore, the above components of the device 100 may communicate via one or more communication buses or signal lines 103 .

[0044] It should be understood that the above-described device 100 is only one example of an electronic device, and that the device 100 may have more or fewer components than shown, may combine two or more components, or alternatively have a different configuration or arrangement of components. Figure 1 The various components shown in the drawings may be implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing and / or application specific integrated circuits.

[0045] In one aspect of the present application, a temporal information aggregation (TIA) module / block can be easily inserted into a neural network, referred to as a baseline neural network, which is suitable for understanding and classifying the content of a video clip and / or audio track. Figure 2 1 shows a schematic diagram of the structure of a baseline neural network 200 for classifying and analyzing video clips proposed in an embodiment of the present application. The neural network 200 can be implemented in the electronic device 100 as an instruction set and / or program. Therefore, in order to classify a video clip or an audio track, the chip 104 and / or the processor 120 of the device 100 can communicate with the memory 102 and execute the instruction set of the neural network 200 to load and activate the neural network 200. Thus, the neural network 200 can perform the actions and / or methods disclosed in the present application.

[0046] In addition, for illustrative purposes only, the following description uses a 3-D ResNet-50 deep learning neural network as an example of a neural network 200 to classify video clips. However, those skilled in the art should understand that other types of two-dimensional or three-dimensional neural networks can also be used for video content understanding (e.g., video classification). In addition, in addition to video content understanding, in the scenario of audio signal understanding (such as audio classification), the TIA module / block can also be inserted into a neural network suitable for audio content understanding. Any type of two-dimensional or three-dimensional neural network suitable for audio track content learning and / or classification, such as a two-dimensional ResNet-50, can be used in this application.

[0047] The baseline network 200 may include multiple stages connected in series to each other. Each stage may include multiple blocks. The baseline network 200 may repeatedly adopt / adopt 3 baseline neural network block structures: structure A, structure A' and structure B.

[0048] Figure 3A A schematic block diagram of structure A proposed in an embodiment of the present application is shown. The neural network block structure A may include multiple separable three-dimensional convolutional layers, that is, the structure A may be designed / constructed to perform decoupled convolution operations on the input in different domains of different layers. For example, for an input feature map 302 related to multiple original image frames of a video clip, the feature map 302 may include information in the spatial domain and the temporal domain. Accordingly, a feature map can be represented by TxHxWxC, where T represents the time axis, H and W represent the spatial axes respectively, and C is the number of channels of the feature map. Specifically, for structure A, the feature map 302 can be expressed as TxHxWxC=T A xH A xW A xC A , where the subscript A represents structure A. Figure 3A The neural network block structure A in the example may be designed to perform spatial convolution and temporal convolution on the input feature map 302 through independent convolution layers. To this end, the neural network block structure A may include 4 convolution layers.

[0049] The first convolutional layer 312 may be designed / constructed to convert the number of channels of the input feature map from C to A Change to C A / 4. To this end, the first convolution layer 312 may include a kernel size of 1x1x1 and a stride of 1x1x1 on the T, H, and W axes of the input feature map 302. Accordingly, the output feature of the convolution layer 312 may be T A xH A xW A xC A / 4 size.

[0050] The second convolutional layer 314 may be configured to perform and / or perform pure spatial convolution on the output of the first convolutional layer 312 without performing any operation in the time domain. To this end, the second convolutional layer 314 may be designed / constructed to perform convolution in the spatial domain, i.e., the kernel size may be 1 in the time axis (T axis) and greater than 1 in the spatial domain (H axis and W axis). Figure 3AIn the illustrated embodiment, the kernel size of the second convolutional layer 314 is 1x3x3. In addition, the second convolutional layer 314 can be designed / constructed to keep the size of the feature map unchanged in the spatial domain. Therefore, the second convolutional layer 314 can additionally include a padding operation on the spatial domain, and the span in the second convolutional layer 314 can be 1x1x1. The second convolutional layer 314 can also be designed / constructed to keep the number of channels unchanged in the output of the first convolutional layer 312. Accordingly, the size of the output of the second convolutional layer 314 can be T A xH A xW A xC A / 4.

[0051] The third convolutional layer 316 may be designed / constructed to perform and / or perform pure temporal convolution on the output of the second convolutional layer 314 without performing any operation in the spatial domain. To this end, the third convolutional layer 316 may be designed / constructed to perform convolution in the time domain, i.e., the kernel size may be 1x1 in the spatial axis (H axis and W axis) and greater than 1 in the time domain (T axis). Figure 3A In the illustrated embodiment, the kernel size of the third convolutional layer 316 is 3x1x1. In addition, the third convolutional layer 316 can be designed / constructed to keep the size of the feature map unchanged in the time domain. Therefore, the third convolutional layer 316 can additionally include a padding operation in the time domain, and the span in the third convolutional layer 316 can be 1x1x1. The third convolutional layer 316 can also be designed / constructed to keep the number of channels in the output of the first convolutional layer 314 unchanged. Therefore, the size of the output of the second convolutional layer 316 can be T A xH A xW A xC A / 4.

[0052] The fourth convolutional layer 318 is designed / constructed to convert the number of channels of the input feature map from the output feature map of the third convolutional layer 316 from C to A / 4Change back to C A To this end, the fourth convolutional layer 318 may include a kernel size of 1x1x1 and a stride of 1x1x1 on the T, H, and W axes. Therefore, the output feature map of the fourth convolutional layer 318 may be T A xH A xW A xC A The size of is the same as the input feature map 302. The output feature map of the fourth convolutional layer 318 can be called a residual feature map.

[0053] After obtaining the residual feature map, the structure A may further include a summing unit 319 to perform summing 319 between the original input 302 and the residual feature map. The summing result may be the output feature map 303 of the structure A.

[0054] In addition, structure A can also include Figure 3A Several rectifiers in the positions shown.

[0055] Figure 3B : A schematic block diagram of the structure A' proposed in an embodiment of the present application is shown. The neural network block structure A' may include multiple separable three-dimensional convolutional layers, that is, the structure A' may be configured / designed to perform decoupled convolutions on the input in different domains of different layers. For example, for an input feature map 302' related to multiple original image frames of a video clip, the feature map 302' may include information in the spatial domain and the temporal domain. Accordingly, the feature map 302' may be represented by TxHxWxC=T A’ xH A’ xW A’ xCx represents, where the subscript A' refers to the structure A', and Cx refers to the number of channels of the input feature map. Figure 3B The neural network block structure A' in can be designed / constructed to perform spatial convolution and temporal convolution on the input feature map 302' through separate and independent convolution layers. To this end, the neural network block structure A' can include 4 convolution layers.

[0056] The first convolutional layer 312' can be designed / constructed to output any number of input channels Cx from the feature map 302' to a predetermined number CA' of channels of the input feature map. To this end, the first convolutional layer 312' can include a kernel size of 1x1x1 and a stride of 1x1x1 on the T, H, and W axes. Therefore, the output features of the convolutional layer 312' can be T A’ xH A’ xW A’ xC A’ size.

[0057] The parameters in the second convolution layer 314', the third convolution layer 316', and the fourth convolution layer 316' of the structure A' and the rectifier ReLu can remain the same as the corresponding layers of the structure A. In addition, the summation unit 319' of the structure A' can synthesize the residual feature map (i.e., the output from the fourth convolution layer 318') and the output feature map of the first convolution layer 312'. Accordingly, the channels of the final output feature map 304' of the structure A' can be 4C A’ Therefore, the final output feature map 304' can be T A’ xH A’ xW A’ x4C A’ size.

[0058] Figure 3C: shows a schematic block diagram of the structure B proposed in the embodiment of the present application. The neural network block structure B may include multiple separable three-dimensional convolutional layers, that is, the structure B may be configured / designed to perform decoupled convolution on the input in different domains of different layers. For example, for an input feature map 306 related to multiple original image frames of a video clip, the feature map 306 may include information in the spatial domain and the temporal domain. Therefore, the feature map 306 may be represented by TxHxWxC=T B xH B xW B xC B Represents, where the subscript B represents structure B. Figure 3C The neural network block structure B in the example may be designed to perform spatial convolution and temporal convolution on the input feature map 306 through independent convolution layers. The neural network block structure B may also reduce the size of the output feature map. To this end, the neural network block structure B may include 5 convolution layers.

[0059] The first convolutional layer 322 may be designed / constructed to convert the number of channels of the input feature map from C in the feature map 306 to B Reduce to C B / 4. To this end, the first convolutional layer 322 may include a kernel size of 1x1x1 and a stride of 1x1x1 on the T, H, and W axes. Accordingly, the output features of the convolutional layer 322 may be T B xH B xW B xC B / 4 size.

[0060] The second convolutional layer 314 may be configured to perform and / or perform pure spatial convolution on the output of the first convolutional layer 322 without performing any operation in the time domain. To this end, the second convolutional layer 324 may be designed / constructed to perform convolution in the spatial domain, i.e., the kernel size may be 1 in the time axis (T axis) and greater than 1 in the spatial domain (H axis and W axis). Figure 3A In the illustrated embodiment, the kernel size of the second convolutional layer 314 is 1x3x3. In addition, the second convolutional layer 314 can be designed / constructed to reduce the size of the feature map in the spatial domain. Accordingly, the span of the second convolutional layer 324 can be 1x2x2. The second convolutional layer 314 can also be designed / constructed to keep the number of channels unchanged in the output of the first convolutional layer 322. Accordingly, the size of the output of the second convolutional layer 314 can be T B xH B / 2xW B / 2xC B / 4.

[0061] The third convolutional layer 326 may be designed / constructed to perform and / or perform pure temporal convolution on the output of the second convolutional layer 324 without doing anything in the spatial domain. To this end, the third convolutional layer 326 may be designed / constructed to perform convolution in the time domain, i.e., the kernel size may be 1x1 in the spatial axis (H axis and W axis) and greater than 1 in the time domain (T axis). Figure 3C In the illustrated embodiment, the kernel size of the third convolutional layer 326 is 3x1x1. In addition, the third convolutional layer 326 can be designed / constructed to reduce the size of the feature map in the time domain. For example, the span of the third convolutional layer 326 can be 2x1x1. The third convolutional layer 326 can be designed / constructed to keep the number of channels unchanged in the output of the second convolutional layer 324. Accordingly, the size of the output of the second convolutional layer 326 can be T B / 2xH B / 2xW B / 2xC B / 4.

[0062] The fourth convolutional layer 328 may be designed / constructed to convert the number of channels of the input feature map from the output feature map of the third convolutional layer 326 from C to B / 4Change back to C B To this end, the fourth convolutional layer 328 may include a kernel size of 1x1x1 and a stride of 1x1x1 on the T, H, and W axes. Therefore, the output features of the convolutional layer 328 may be T B / 2xH B / 2xW B / 2xC B The output feature map of the fourth convolutional layer 328 can be called the residual feature map of structure B.

[0063] The fifth convolutional layer 320 can be designed / constructed to process the input feature map 306 so that the output of the fifth convolutional layer matches the size of the residual feature map, that is, the size of the input feature map is adjusted to T without changing the number of channels of the input feature map of the feature map 306. B / 2xH B / 2xW B / 2. To this end, the fifth convolutional layer 320 may include a kernel size of 1x1x1 and a stride of 2x2x2 on the T, H, and W axes. Accordingly, the output features of the convolutional layer 322 may be T B / 2xH B / 2xW B / 2xC B size.

[0064] After obtaining the residual feature map, the structure B may further include a summing unit 329 to sum the output feature map of the fifth convolutional layer 320 and the residual feature map. The summing result may be the output feature map 308 of the structure B.

[0065] In addition, structure B can also include Figure 3C Several rectifiers in the positions shown.

[0066] refer to Figure 2 , the baseline network 200 may include multiple stages connected in series. Each stage may include multiple blocks. For example, Figure 2 In the illustrated embodiment, the baseline network 200 may include 6 stages, namely, a first stage 210, a second stage 220, a third stage 230, a fourth stage 240, a fifth stage 250, and a sixth stage 260. Specifically, for the ResNet neural network, the second stage 220 (Res2), the third stage 230 (Res3), the fourth stage 240 (Res4), and the fifth stage 250 (Res5) may include a plurality of residual blocks, such as Figure 3A -C.

[0067] The first stage 210 may be the first neural network stage that receives input. Figure 2 In the embodiment shown in , the first stage 210 can receive an input 202. The input 202 can be a plurality of raw image frames of a video clip. The plurality of image frames can be constructed in the same order as in the video clip, that is, the image frames are arranged one after another in time. The input can also be an output feature map of another video processing system, corresponding to the plurality of raw image frames. Because the input 202 is or corresponds to the plurality of raw image frames of the video clip, the input 202 can be a three-dimensional input feature map, whose size is TxHxWxC=T 0 xH 0 xW 0 xC 0 , where C 0 is the number of channels in the input. W 0 and H 0 are the width and height of input 202, corresponding to the width and height of each image frame in the video clip. 0 is the length of the input feature data 202, corresponding to the time of the video clip. For example, if the video plays 32 color frames per second, each image has a resolution of 112x112 and is presented in the form of a superposition of red, green, and blue color spaces (i.e., 3 channels), then a 1-second video clip includes a sequence of 32 image frames, and the dimension of the image frame sequence as the input 202 of the neural network 200 is 32x112x112x3.

[0068] The first stage 210 can take an input 202 and output a feature map 216. The first stage 210 can include several convolutional layers that extract spatial features and temporal features from the input 202, respectively, and adjust the number of channels of the input 202 to a predetermined number. For example, the first stage 210 can include a spatial convolutional layer 212 and a temporal convolutional layer 214 connected in series to the spatial convolutional layer 212. For example, the spatial convolutional layer 212 can be a 1x7x7 layer with a span of 1x2x2 on the T, H, and W axes, respectively, and an output channel size of 64, that is, the output includes 64 channels; and the temporal convolutional layer 214 can be a 3x1x1 layer with a span of 2x11 on the T, H, and W axes, respectively, and an output channel C=64.

[0069] Accordingly, the spatial convolution layer 212 may receive the input 202 of size TxHxWxC=32x112x112x3 and perform convolution on the input 202 in the spatial domain (i.e., along the H axis and the W axis). The output of the spatial convolution layer 212 may be a feature map of size TxHxWxC=32x56x56x64, including the extracted spatial features of the input 202. The output of the spatial convolution layer 212 may then be input to the temporal convolution layer 214.

[0070] The temporal convolution layer 214 may be designed / constructed to convolve the output of the convolution layer 202 in the time domain (ie, along the T axis). The output 216 from the temporal convolution layer 214 may be a 16x56x56x64 feature map including both spatial and temporal information of the input 202.

[0071] Because the convolution layer 212 is designed to perform only spatial convolution on the 4-D input, and the convolution layer 214 connected in series with the convolution layer 212 is designed to perform only temporal convolution on the 4-D input, the first stage 210 is a separable 3-D convolution stage.

[0072] The feature map 216 may pass through the second stage 220, the third stage 230, the fourth stage 240, and the fifth stage 250 in sequence. Each of the second to fifth stages includes a plurality of convolution blocks. Each convolution block may include convolution layers of different structures. Optionally, the structure of each convolution block may be the same.

[0073] For example, in Figure 2 In the example, the second stage 220 may include three convolution blocks. The structures of the three convolution blocks 222, 224, and 226 may be respectively adopted as [A', A, B]. Specifically, the convolution block 222 may be a convolution block of structure A', where the number of channels C A’ =64; convolution block 224 may be a structure A convolution block; and convolution block 226 may be a structure B block. Accordingly, for a 32x112x112x3 original image frame, the output feature map of the second stage 220 may be 8x28x28x256.

[0074] The third stage 230 may include four convolution blocks. The structures of the four convolution blocks 232, 234, 236, and 238 may be respectively adopted as [A', A, A, B]. Specifically, the convolution block 232 may be a convolution block of structure A', where the number of channels C A’ =128; convolution block 234 and convolution block 236 may be structure A convolution blocks respectively; and convolution block 236 may be structure B block. Therefore, for a 32x112x112x3 original image frame, the size of the output feature map of the third stage 230 may be 4x14x14x512.

[0075] The fourth stage 240 may include 6 convolution blocks. The structures of the 6 convolution blocks 242, 243, 244, 245, 246 and 247 may be respectively adopted as [A', A, A, A, A, B]. Specifically, the convolution block 242 may be a convolution block of structure A', where the number of channels C A’ =256; convolution block 243, convolution block 244, convolution block 245 and convolution block 246 can be structure A convolution blocks respectively; and convolution block 237 can be a structure B block. Therefore, for a 32x112x112x3 original image frame, the size of the output feature map of the fourth stage 240 can be 2x7x7x1024.

[0076] The fifth stage 250 may include three convolution blocks. The structures of the three convolution blocks 252, 254, and 256 may be respectively adopted as [A', A, A]. Specifically, the convolution block 252 may be a convolution block of structure A', where the number of channels C A’ =512; convolution block 254 may be a structure A convolution block; and convolution block 256 may be a structure A block. Accordingly, for a 32x112x112x3 original image frame, the size of the output feature map of the fifth stage 250 may be 2x7x7x2048.

[0077] The sixth stage 260 may include two layers. Layer 262 may be a global average pooling layer, and layer 264 may be a full synthesis layer. Through the sixth stage 260, for a 32x112x112 original image frame, the output feature map of the sixth stage 260 may be 1x1x1x2048.

[0078] Although the 3D separable baseline neural network 200 includes temporal convolution operations in several neural network layers, the 3D temporal convolution operations only extract limited temporal information from the input sequence of image frames 202. For example, Figure 2In the illustrated embodiment, the temporal convolution operation operates on only three frames of feature maps in the neighborhood at a time. The time corresponding to these three frames may not be sufficient to understand, classify, and analyze the semantics of the video clip content relative to time. It is necessary to modify the three-dimensional separable baseline neural network 200 to extract temporal information at multiple and larger time scales and supplement the temporal information into the extracted feature maps.

[0079] Figure 4 FIG. 4 is a schematic diagram showing the structure of a modified neural network 400 for classifying video clips proposed in an embodiment of the present application. Figure 2 In addition to the elements of the 3D baseline neural network 200 shown, the modified neural network 400 further includes a plurality of temporal information aggregation (TIA) modules / blocks 500 to extract temporal information of the input video clip image frames 202 having different temporal scales. Figure 4 , the modified neural network 400 may include one TIA block in each of the other blocks of the third stage 230 and the fourth stage 240, that is, the modified neural network 400 may insert two TIA blocks 500 in the third stage 230 and three TIA blocks 500 in the fourth stage. However, the TIA block 500 may be inserted after any convolutional layer and help extract temporal information at different time scales.

[0080] Figure 5 FIG. 5 is a schematic diagram of a time information aggregation block 500 proposed in an embodiment of the present application. As described above, the TIA block 500 can be connected in series between two baseline neural network blocks.

[0081] The baseline neural network block ("first baseline neural network block") before the TIA block may be designed and / or constructed as a three-dimensional separable neural network block that converts a first feature map into a second feature map. The first feature map may be a feature map of a plurality of image frames 202. Further, the baseline neural network block may independently extract one or more spatial features and one or more temporal features through different neural network layers to convert the first feature map into a second feature map.

[0082] The TIA block (also called "target neural network block") 500 can be adjacent to and in series with the first baseline neural network block. In addition, the TIA block 500 can be designed and / or constructed to process the second feature map 502 in the time domain and generate a third feature map 516.

[0083] To this end, the TIA block 500 may include a pooling unit. The pooling unit may include n pooling layers, which are designed and / or constructed to temporarily pool the second feature map 502 into n first intermediate feature maps {504-i} using n pooling windows of different sizes, respectively, where i=1, 2, ..., n,. Here, n is a predetermined positive integer. For example, in this embodiment, the size of the second feature map 502 may be T t xH t xW t xC t , where the subscript t represents the target neural network block, i.e., the TIA block. In order to effectively combine multi-scale temporal information, temporal pooling of various scales is adopted to capture the temporal information of the second feature map 502 of different resolutions. For example, in Figure 5 In the embodiment of the present invention, the TIA block 500 may include 3 pooling layers (n=3), each layer independently using a time pooling window. In order to extract different time information from the baseline neural network 200, the size of each pooling window may be different from the kernel size of the baseline neural network 200 in the time domain. Specifically, since the time information extracted by the baseline neural network 200 corresponds to a short time period, the size of all n pooling windows may be larger than the kernel size of the baseline neural network 200 with respect to the time domain. In addition, the size of the n pooling windows may correspond to the time length, wherein the modified neural network 400 may be designed and / or designed to understand the time information.

[0084] For example, the first pooling window can adopt T t Accordingly, the first feature map 502-1 can be temporally pooled to the first intermediate feature map 504-1 (size 1xH t xW t xC t ). The second pooling window can be T t / 2. Therefore, the second feature map 502-2 can be temporally pooled to the first intermediate feature map 504-2 (with a size of 2xH t xW t xC t ). The third pooling window size can be T t / 4. Accordingly, the third feature map 502-3 can be temporally pooled to the first intermediate feature map 504-3 (size 4xH t xW t xC t ).

[0085] The TIA block 500 may rescale the temporally subsampled feature maps (e.g., the second intermediate feature maps 504-1, 504-2, and 504-3) to a predetermined size, respectively, and temporally, to generate a second intermediate feature map 506. In some embodiments, the predetermined size may be the original size of the second feature map 502, i.e., T t xH t xW t xC t For example, in Figure 5 , the first intermediate feature map 504-1 can be rescaled to T t xH t xW t xC t , generating a second intermediate feature map 506-1; the first intermediate feature map 504-2 can be rescaled to T t xH t xW t xC t , generating a second intermediate feature map 506-2; and the first intermediate feature map 504-3 can be rescaled to T t xH t xW t x1, generates a second intermediate feature map 506-3.

[0086] The TIA block 500 may synthesize a second intermediate feature map (eg, 506-1, 506-2, 506-3) along the channel axis of the sub-sampled feature map to generate a second intermediate feature map of size T t xH t xW t xC t (For example, C t =1, n=3). For example, Figure 5 In the embodiment, the TIA block 500 may include a synthesizer 508 to synthesize the second intermediate feature maps 506-1, 506-2, and 506-3 along its channel axis C. Since the size of each second intermediate feature map is T t xH t xW t xC t , the resulting synthesized feature map can be 3 channels, represented by T t xH t xW t x3C t .

[0087] A 1x1x1 convolutional layer 510 may then be applied to the synthesized feature map to produce a residual feature map of a predetermined size. To this end, the TIA block may include a 1x1x1 convolutional layer 510 to change the channel size of the synthesized feature map to the same size as the second feature map 502, i.e., the channel size may be changed from nCt Reduced to C t The resulting feature map can be of size T t xH t xW t xC t The residual feature map 512 of .

[0088] The residual feature map 512 may be added to the input feature map (i.e., the second feature map 502) as residual information to generate an output feature map (i.e., the third feature map 516) of the TIA block 500. According to an embodiment, the TIA block 500 may further include a summing unit 514 to perform element-wise summation between the second feature map 502 and the residual feature map 512.

[0089] The temporal pooling employed by the TIA block 500 may be any suitable pooling algorithm. For example, the TIA block 500 may employ average pooling or maximum pooling. According to an embodiment, the TIA block 500 may utilize maximum pooling. In addition, the rescaling employed in the TIA block 500 may be any type of suitable rescaling algorithm. For example, the TIA block 500 may employ interpolation or deconvolution to rescale the feature map. According to an embodiment, the TIA block 500 may utilize bilinear interpolation to rescale the feature map.

[0090] The TIA block 500 may output the third feature map 516 to the second baseline neural network block. Before modification, the second baseline neural network block may be a block adjacent to the first baseline neural network block in the baseline neural network 200. For example, the second baseline neural network block may be a three-dimensional ResNet network block. The second baseline neural network block may receive the third feature map 516 and convert the third feature map 516 into a fourth feature map by processing at least one spatial feature of the second feature map.

[0091] In various domains, it is often natural and meaningful to represent a data object by a collection of its partial or component features. It can be seen from the above embodiments that the temporal convolution operation in the 3-D separable baseline neural network 200 operates on 3 frames of feature maps of the neighborhood at a time. The time corresponding to these 3 frames may not be sufficient to understand, classify and analyze the temporal semantics of the content of the video clip. However, the TIA block 500 provides temporal pooling for various time scales. By dividing the temporal information (sequence of video frames) into layers from fine to coarse, the temporal patterns are pooled at different scales and local features are aggregated therein. The pooling operation ensures that temporal information of multiple and larger time scales is extracted and the temporal information is supplemented into the extracted feature maps. Therefore, the modified neural network 400 can have a better ability to classify and analyze video clips.

[0092] Fig. 6A-C is a schematic diagram showing the time information that can be extracted by the time information aggregation block 500 proposed in the embodiment of the present application. FIG. 6A to FIG. 6C , the TIA block 500 may store time information (eg, having a size T on the time axis) t The video frame sequence) is divided into finer (T t / 4) to coarser levels (T t ), pooling temporal patterns at variable scales and aggregating local features among them.

[0093] Fig. 6A 612 , 614 , 616 , and 618 .

[0094] Figure 6B is a schematic diagram of the time information that can be extracted by the TIA block 500 when the pooling window size is T / 2. Fig. 6A In contrast, the TIA block 500 may extract the baseball player's movement information 622 , 624 over a longer period of time.

[0095] Figure 6C This illustrates a scenario that the TIA block 500 can extract when the pooling window size is T. Accordingly, the TIA block 500 can extract the motion information 632 of the baseball player in the entire video frame sequence.

[0096] Figure 7 A flowchart of a method for analyzing multiple images constructed in time series by at least one neural network proposed in an embodiment of the present application is shown. The method and at least one neural network can be stored as a set of instructions in the memory 102 (e.g., a non-temporary storage medium, such as a CD-ROM, a hard disk, etc.) of the electronic device 100, such as Figure 1 When executing the method, the processor 120 of the electronic device 100 can communicate with the memory 102, and then perform the action and / or method through the modified neural network 400. Therefore, it can be combined Figure 4 and Figure 5 and its description to describe the following method.

[0097] In step 702, a first block converts a first feature map associated with a plurality of images into a second feature map by processing at least one spatial feature in the first feature map.

[0098] The conversion of the first feature map can be performed by the first block of the baseline neural network, where the baseline neural network can have the same structure as the neural network 200. Further, the conversion of the first feature map to the second feature map can include processing the first feature map in the time domain using a temporal convolution layer and in the spatial domain using a spatial convolution layer. The temporal convolution layer can include a kernel having a temporal component on the temporal axis. For example, the kernel of the temporal convolution layer can be 3x1x1, where the temporal component can be the number "3".

[0099] Step 704 , the target neural network block converts the second feature map into a temporal feature map (eg, a temporal residual map) by performing temporal processing on the second feature map. For example, the target neural network block may be the TIA block 500 .

[0100] To this end, the target neural network can first pool the second feature map into n intermediate feature maps in time, using n pooling windows of different time scales. Here, n is a predetermined positive integer, such as 3; and the different time scales can be T t 、T t / 2 and T t / 4, where T t is the size of the second feature map along the time axis. Furthermore, the time scale of the n pooling windows is larger than the time component of the kernel. For example, T t / 4 can be larger than the temporal component of the kernel so that the target neural network can capture additional temporal information over multiple video image frames than the temporal convolutional layer of the first block of the baseline neural network.

[0101] The target neural network 500 may rescale each of the n intermediate feature maps to a predetermined size in time. For example, the predetermined size may be the size of the second feature map.

[0102] Then, the target neural network 500 may synthesize the n rescaled intermediate feature maps into a synthesized feature map. The target neural network 500 may apply a 1x1x1 convolutional layer to the synthesized feature map to adjust the size of the synthesized feature map to the size of the second feature map, such as shrinking the channel size of the synthesized feature map to the channel size of the second feature map. The adjusted synthesized feature map may be a temporal feature map, such as a residual feature map of the target neural network.

[0103] In step 706, the target neural network block 500 generates a third feature map by combining the temporal feature map with the second feature map. For example, the combination may be performed by element-wise summing of the temporal residual feature map and the second feature map.

[0104] In step 708 , the target neural network block 500 sends the third feature map as input to the second block of the baseline neural network 200 .

[0105] Having thus described the basic concepts, it may be quite apparent to those skilled in the art after reading this detailed disclosure that the foregoing detailed disclosure is intended to be presented by way of example only and not by way of limitation. Various changes, improvements, and modifications may occur and, although not expressly described herein, are of interest to those skilled in the art. For example, the steps in the method of the present application may not necessarily be operated in the order described. The steps may also be operated in part and / or in other combinations as reasonably contemplated by those skilled in the art. These changes, improvements, and modifications are proposed by the present application and are within the spirit and scope of the exemplary embodiments of the present application.

[0106] In addition, certain terms have been used to describe embodiments of the present application. For example, the terms "one embodiment" and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Therefore, it is emphasized that it should be understood that "one embodiment" or "another embodiment" mentioned twice or more in various parts of this specification does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics can be combined in a suitable manner in one or more embodiments of the present application.

[0107] In addition, those skilled in the art will appreciate that various aspects of the present application may be illustrated and described in any of a number of patentable categories or contexts, including any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Thus, various aspects of the present application may be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in combination with software and hardware, which may be generally referred to as "blocks," "modules," "engines," "units," "components," or "systems." In addition, various aspects of the present application may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0108] A computer readable signal medium may include a propagated data signal having computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including electromagnetic, optical, or the like, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium, and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code embodied on a computer readable signal medium may be transmitted using any suitable medium, including wireless, wired, fiber optic cable, radio frequency, or the like, or any suitable combination of the foregoing.

[0109] The computer program code for performing the operations of various aspects of the present application content can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., traditional procedural programming languages, such as "C" programming language, VisualBasic, Fortran, Perl, COBOL, PHP, ABAP, dynamic programming languages, such as Python, Ruby and Groovy, or other programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., using an Internet service provider via the Internet) or in a cloud computing environment, or provided as a service, such as software as a service (SaaS).

[0110] In addition, the order of the processing elements or sequences described, or the use of numbers, letters or other designations therefor, is not intended to limit the claimed processes and methods to any order unless the claims clearly define it. Although the above disclosure discusses various useful embodiments currently considered to be the contents of the present application through various examples, it should be understood that such detailed description is only for this purpose, and the attached claims are not limited to the disclosed embodiments, but are intended to cover modifications and equivalent arrangements within the spirit and scope of the disclosed embodiments. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a pure software solution. For example, installation on an existing server or mobile device.

[0111] Likewise, it should be understood that in the foregoing description of the embodiments of the present application, various features are sometimes combined in one embodiment, figure, or description thereof in order to simplify the disclosure to aid understanding of one or more different embodiments. However, this disclosure method should not be interpreted as reflecting only an intention that the claimed subject matter requires more features than those explicitly listed in each claim. On the contrary, the claimed subject matter may be less than all the features of the foregoing single disclosed embodiment.

Claims

1. A neural network system, implemented by one or more electronic devices, characterized in that: The method comprises a target neural network block configured to be inserted between a first baseline neural network block and a second baseline neural network block in a baseline neural network; wherein the target neural network block comprises: At least one pooling unit, configured to: Receiving an input feature map output by the first baseline neural network block, wherein the input feature map is output by the first baseline neural network block when processing an image; and Temporally pooling the input feature map into at least one intermediate feature map; and at least one other processing unit, configured to: Temporally processing the at least one intermediate feature map to generate a residual feature map; and The residual feature map is combined with the input feature map to generate an output feature map for input into the second baseline neural network block.

2. The neural network system according to claim 1, characterized in that The at least one pooling unit is used to pool the input feature map into n intermediate feature maps through n pooling windows of different time scales; wherein n is a predetermined positive integer greater than 1.

3. The neural network system according to claim 2, characterized in that Further including: The baseline neural network is used to process multiple images constructed in time series; Among them, the first baseline neural network block includes one of at least one temporal convolution layer or at least one spatial convolution layer; the at least one temporal convolution layer is used to process the first feature map in the time domain, and the at least one spatial convolution layer is used to process the first feature map in the spatial domain; the first feature map is a feature map related to the multiple images, and the first feature map includes information in the spatial domain and the temporal domain.

4. The neural network system according to claim 3, characterized in that The kernel of the at least one temporal convolutional layer comprises a temporal component.

5. The neural network system according to claim 2, characterized in that: The target neural network block is further used to temporally rescale each of the n intermediate feature maps to a predetermined size to generate the residual feature map.

6. The neural network system according to claim 5, characterized in that: In order to generate the residual feature map, the at least one other processing unit further comprises: a synthesizer, configured to synthesize the n rescaled intermediate feature maps to obtain a synthesized feature map; and A layer is used to temporally rescale the synthesized feature map to the size of the output feature map to generate the residual feature map.

7. The neural network system according to claim 6, characterized in that The target neural network block is used to synthesize the n rescaled intermediate feature maps along their respective channel axes, thereby synthesizing the n rescaled intermediate feature maps to obtain the synthesized feature map.

8. The neural network system according to claim 7, characterized in that: The layer further comprises a convolutional layer applied to the synthesized feature map to reduce the channel size of the synthesized feature map to the channel size of the second feature map, thereby temporally rescaling the synthesized feature map; The second feature map is the input feature map.

9. The neural network system according to claim 1, characterized in that: The at least one other processing unit comprises a summing unit for element-wise summing of the residual feature map and the input feature map, thereby combining the residual feature map with the input feature map.

10. The neural network system according to claim 1, characterized in that: The second baseline neural network block is connected in series with the target neural network block; the second baseline neural network block is used to: receiving the output feature map as a third feature map; and At least one of the spatial features or the temporal features of the third feature map is processed to convert the third feature map into a fourth feature map.

11. The neural network system according to claim 1, characterized in that: The first baseline neural network adopts a three-dimensional ResNet network structure.

12. A method, characterized in that A plurality of images constructed by analyzing a time series using at least one neural network; wherein: Receiving an input feature map output by a first baseline neural network block in a baseline neural network, wherein the input feature map is output by the first baseline neural network block after processing an image; Temporally pooling the input feature map into at least one intermediate feature map; Temporally processing the at least one intermediate feature map to generate a residual feature map; and The residual feature map is combined with the input feature map to generate an output feature map for input into a second baseline neural network block in the baseline neural network.

13. The method according to claim 12, characterized in that Pooling the input feature map into at least one intermediate feature map includes: temporally pooling the input feature map into n intermediate feature maps through n pooling windows of different time scales; wherein n is a predetermined positive integer greater than 1.

14. The method according to claim 13, characterized in that Further including: The first feature map is processed in at least one of a time domain using a temporal convolution layer or a spatial domain using a spatial convolution layer to convert the first feature map into a second feature map as the input feature map; the second feature map is the input feature map; the first feature map is a feature map related to the multiple images, and the first feature map includes information in the spatial domain and the temporal domain.

15. The method according to claim 14, characterized in that The kernel of the temporal convolutional layer includes a temporal component.

16. The method according to claim 13, characterized in that The generating the residual feature map includes: temporally rescaling each of the n intermediate feature maps to a predetermined size.

17. The method according to claim 16, characterized in that The generating the residual feature map further comprises: synthesizing the n rescaled intermediate feature maps to obtain a synthesized feature map; and The synthesized feature map is temporally rescaled to the size of the output feature map to generate the residual feature map.

18. The method according to claim 17, characterized in that The temporally rescaling the synthesized feature map comprises applying a convolutional layer to the synthesized feature map to adjust the size of the synthesized feature map to the size of the input feature map.

19. The method according to claim 18, characterized in that The temporally rescaling the synthesized feature map comprises: applying the convolutional layer to the synthesized feature map to reduce the channel size of the synthesized feature map to the channel size of a second feature map; the second feature map is the input feature map.

20. The method according to claim 14, characterized in that The combining the residual feature map and the input feature map includes: element-wise summing the residual feature map and the input feature map.

Images