Fast-slow channel information fusion method, device, storage medium and program product

Abstract

The present disclosure relates to the technical field of artificial intelligence, and particularly provides a fast / slow lane information fusion method, device, storage medium and program product. The method obtains first lane features and second lane features in fast / slow networks, reduces dimensions or expands frames of the first lane features, makes the frame number dimensions of the first lane features and the second lane features same, obtains first temporary lane features, performs a pooling operation on the first temporary lane features, and obtains second temporary lane features; generates a feature map based on the second temporary lane features; obtains new first lane features based on the feature map, the first lane features and the second lane features; and processes the new first lane features by using the first lane. The embodiment realizes fusion of the first lane features and the second lane features, effectively interacts information corresponding to the slow and fast lanes in two directions, improves feature effectiveness, and reduces redundant information.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium, and program product for fast and slow channel information fusion. Background Technology

[0002] With the widespread use of surveillance cameras, they have played an important role in maintaining social order and collecting legal evidence. However, due to the mismatch between the number of surveillance cameras, the amount of video footage, and the number of surveillance personnel, a large amount of surveillance video cannot be processed in a timely manner, and real-time criminal acts are difficult to be effectively detected and dealt with in the first instance. Therefore, it is very important to carry out intelligent automatic processing of surveillance video and automatic notification and alarm.

[0003] Compared to images, videos possess a time dimension, containing more complex and difficult-to-extract information. Furthermore, videos contain numerous interactive behaviors; the same action can be completely different in different scenarios. For example, chopping vegetables with a knife in the kitchen is clearly normal behavior, but doing so outdoors is obviously abnormal. Given the variety of abnormal behaviors and their relatively short duration, effectively extracting common features from videos and quickly training models for specific scenarios is the main challenge.

[0004] The video abnormal behavior recognition network is an important component of the intelligent monitoring and alarm system. It can effectively identify abnormal behaviors in monitoring videos and send alarm information in a timely manner. The general process is shown in the figure below. First, the video stream is input. Since the frame rate of various monitoring devices is relatively low and the information difference within adjacent frames is small, key frames are extracted. Then, the key frames are sent to the corresponding deep feature extraction network to obtain efficient deep features. After that, the abnormal behavior prediction score is obtained through the prediction module. If it exceeds the threshold, alarm measures are activated (such as sending alarm SMS).

[0005] The aforementioned deep learning-based video anomaly recognition method is a commonly used solution for video anomaly recognition; however, existing methods have certain shortcomings: Existing deep learning networks do not effectively utilize the correlation between feature information from long-span frames and feature information from short-span frames, and do not effectively fuse the two types of information, making it difficult to effectively obtain and fuse the respective advantages of fast and slow channels. Summary of the Invention

[0006] This disclosure is made in view of the above-mentioned problems. This disclosure provides a method, apparatus, storage medium, and program product for fusing fast and slow channel information.

[0007] According to the first aspect of this disclosure, a method for fusing fast and slow channel information is provided, comprising: Obtain the first channel feature and the second channel feature in the fast and slow network. The first channel feature is the feature output by the convolutional layer of the first channel in the fast and slow network. The second channel feature is the feature output by the convolutional layer corresponding to the convolutional layer in the second channel of the fast and slow network. The first channel and the second channel are different. The first channel is either the fast channel or the slow channel, and the second channel is either the fast channel or the slow channel. The first channel feature is reduced in dimension or expanded in frame so that the first channel feature and the second channel feature have the same frame number dimension, thus obtaining the first temporary channel feature; The first temporary channel feature is pooled to obtain the second temporary channel feature; A feature map is generated based on the features of the second temporary channel; Based on the feature map, the first channel feature, and the second channel feature, a new first channel feature is obtained; The new first channel features are processed using the first channel.

[0008] Furthermore, according to the fast and slow channel information fusion method of the first aspect of this disclosure, the first temporary channel features are pooled to obtain the second temporary channel features, including: The first temporary channel feature is subjected to average pooling to obtain average pooled feature; and the first temporary channel feature is subjected to max pooling to obtain max pooled feature. The average pooling feature and the max pooling feature are concatenated to obtain the second temporary channel feature.

[0009] Furthermore, according to the fast and slow channel information fusion method of the first aspect of this disclosure, a feature map is generated based on the features of the second temporary channel, including: When the first channel is a fast channel, the features of the second temporary channel are subjected to 2D convolution processing to obtain the feature map; When the first channel is a slow channel, the second temporary channel features are fully connected to obtain the feature map.

[0010] Furthermore, according to the fast and slow channel information fusion method of the first aspect of this disclosure, a new first channel feature is obtained based on the feature map, the first channel feature, and the second channel feature, including: Obtain the number of video frames processed in one pass of the slow channel; For the given number of frames, the feature map is split to obtain a split feature map; Based on the feature map of the second channel features and the split feature map, the features of the video frame are obtained; The features of the video frame are added to the features of the first channel to obtain the new features of the first channel.

[0011] Furthermore, according to the fast and slow channel information fusion method of the first aspect of this disclosure, the training process of the fast and slow network includes pre-training and scenario-specific training; In this process, after the pre-training is completed, all parameters in the fast and slow networks are retained except for the last M fully connected layers. During the scenario-specific training process, in the first N training iterations, the last M fully connected layers in the fast and slow network are trained. In the remaining training iterations, the last M fully connected layers and layers M-1 to MT are trained, and the learning rate of layers M-1 to MT is less than the learning rate of the last M layers.

[0012] Furthermore, according to the fast and slow channel information fusion method of the first aspect of this disclosure, M is 1 and T is 2.

[0013] According to a second aspect of this disclosure, a fast and slow channel information fusion device is provided, comprising: The acquisition module is used to acquire the first channel features and the second channel features in the fast and slow network. The first channel features are the features output by the convolutional layer of the first channel in the fast and slow network, and the second channel features are the features output by the convolutional layer corresponding to the convolutional layer in the second channel of the fast and slow network. The first channel and the second channel are different. The first channel is either the fast channel or the slow channel, and the second channel is either the fast channel or the slow channel. A dimensionality reduction or frame expansion module is used to reduce the dimensionality or expand the frame of the first channel feature so that the frame number dimension of the first channel feature and the second channel feature are the same, thereby obtaining a first temporary channel feature. A pooling module is used to perform a pooling operation on the first temporary channel feature to obtain the second temporary channel feature; The generation module is used to generate a feature map based on the features of the second temporary channel; The acquisition module is used to obtain a new first channel feature based on the feature map, the first channel feature, and the second channel feature; The processing module is used to process the new first channel feature using the first channel.

[0014] According to a third aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method described in the first aspect. According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program / instructions stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect. According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] As will be described in detail below, the fast and slow channel information fusion method according to the embodiments of this disclosure generates new first channel features for the first channel features and second channel features in the fast and slow network through steps such as dimension processing and pooling operations, thereby realizing the fusion of the first channel features and second channel features, effectively bidirectionally interacting the information corresponding to the slow and fast channels, improving feature effectiveness, and reducing redundant information.

[0016] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 This is a flowchart illustrating a fast and slow channel information fusion method according to an embodiment of the present disclosure.

[0019] Figure 2 This is a schematic diagram illustrating the application of a fast and slow network according to an embodiment of the present disclosure.

[0020] Figure 3 This is a schematic diagram illustrating a prediction and identification module of a fast and slow network according to an embodiment of the present disclosure.

[0021] Figure 4 This is a schematic diagram illustrating the fusion of fast and slow channel information according to an embodiment of this disclosure.

[0022] Figure 5 This is a schematic diagram illustrating the fusion of fast and slow channel information according to an embodiment of this disclosure.

[0023] Figure 6 This is a schematic diagram illustrating a fast and slow channel information fusion device according to an embodiment of the present disclosure.

[0024] Figure 7 This is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0025] Figure 8 This is a schematic diagram illustrating a computer program product according to an embodiment of the present disclosure. Detailed Implementation

[0026] The technical methods of the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings.

[0027] To facilitate understanding of this embodiment, a fast and slow channel information fusion method disclosed in this disclosure will first be described in detail. The execution subject of the fast and slow channel information fusion method provided in this disclosure is generally an electronic device with a certain computing power, such as a terminal device, a server, or other processing device. In some possible implementations, the fast and slow channel information fusion method can be implemented by a processor calling computer-readable instructions stored in memory.

[0028] See Figure 1 The diagram shown is a flowchart of a fast / slow channel information fusion method provided in this embodiment of the present disclosure. The method includes the following steps: Step 101: Obtain the first channel features and the second channel features in the fast and slow networks.

[0029] The first channel feature is the feature output of the first channel convolutional layer in the FastSlow network, and the second channel feature is the feature output of the convolutional layer corresponding to the convolutional layer in the second channel of the FastSlow network. The first channel and the second channel are different; the first channel is either a fast channel or a slow channel, and the second channel is either a fast channel or a slow channel.

[0030] In this embodiment, when the first channel is a fast channel and the second channel is a slow channel, the first channel exhibits fast channel characteristics and the second channel exhibits slow channel characteristics. Similarly, when the first channel is a slow channel and the second channel is a fast channel, the first channel exhibits slow channel characteristics and the second channel exhibits fast channel characteristics.

[0031] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a fast / slow network provided in an embodiment of this application. Figure 2 As shown, the fast and slow network includes a fast channel and a slow channel, and each of the fast and slow channels includes a convolutional layer with corresponding settings.

[0032] Step 102: Reduce the dimensionality or expand the frame of the first channel feature so that the frame number dimension of the first channel feature and the second channel feature are the same, and obtain the first temporary channel feature.

[0033] In this embodiment, when the first channel feature is a fast channel feature, dimensionality reduction processing is required for the first channel feature. Optionally, dimensionality reduction of the fast channel feature can be performed using 3D convolution. When the first channel feature is a slow channel feature, frame expansion is required for the slow channel feature.

[0034] Step 103: Perform a pooling operation on the first temporary channel features to obtain the second temporary channel features.

[0035] In one or more alternative embodiments, the first temporary channel feature is pooled to obtain the second temporary channel feature, including: The first temporary channel features are subjected to average pooling to obtain average pooled features; and the first temporary channel features are subjected to max pooling to obtain max pooled features. The average pooling feature and the max pooling feature are concatenated to obtain the second temporary channel feature.

[0036] It should be understood that pooling should be performed on both fast-channel and slow-channel features according to the above pooling operation process. That is, when pooling fast-channel features, average pooling and max pooling operations are performed separately to obtain average-pooled and max-pooled features. These average-pooled and max-pooled features are then concatenated to obtain the second temporary channel feature of the fast-channel features. Similarly, when pooling slow-channel features, average pooling and max pooling operations are performed separately to obtain average-pooled and max-pooled features. These average-pooled and max-pooled features are then concatenated to obtain the second temporary channel feature of the slow-channel features.

[0037] Step 104: Generate a feature map based on the features of the second temporary channel.

[0038] In this embodiment, feature maps are generated using different methods based on whether the features of the second temporary channel originate from slow channel features or fast channel features.

[0039] Specifically, when the first channel is a fast channel, the features of the second temporary channel are processed by 2D convolution to obtain a feature map; When the first channel is a slow channel, the features of the second temporary channel are processed by full connection to obtain a feature map.

[0040] Step 105: Based on the feature map, the first channel feature, and the second channel feature, obtain the new first channel feature.

[0041] In one or more alternative embodiments, obtaining a new first channel feature based on the feature map, the first channel feature, and the second channel feature may include the following steps: Get the number of video frames processed in one pass of the slow channel; For each frame, the feature map is split to obtain the split feature map; Based on the feature map and split feature map of the second channel feature, the features of the video frame are obtained; The features of the video frame are added to the features of the first channel to obtain new features of the first channel.

[0042] In this embodiment, when obtaining the features of a video frame based on the feature map of the second channel features and the split feature map, the vector corresponding to the feature map of the second channel features is multiplied with the vector corresponding to the split feature map to obtain the features of the video frame.

[0043] Step 106: Process the new first channel features using the first channel.

[0044] In the solution provided in this embodiment, for the first channel features and the second channel features in the fast and slow networks, a new first channel feature is generated through steps such as dimension processing and pooling operations. This achieves the fusion of the first channel features and the second channel features, effectively enabling bidirectional interaction of information corresponding to the slow and fast channels, improving feature effectiveness, and reducing redundant information.

[0045] In related technologies, most video abnormal behavior recognition methods use simple three-layer fully connected recognition networks to obtain a single video abnormal behavior recognition model. This model is used for different abnormal behaviors, resulting in insufficient specialization. Furthermore, when new abnormal behavior categories are introduced, it is difficult to achieve excellent recognition results through simple training. This application proposes a multi-branch prediction network structure and training method to generate specialized models for specific application scenarios, thereby improving the accuracy of abnormal behavior recognition. The specific training method is shown below.

[0046] The training process for fast and slow networks includes pre-training and scenario-specific training; In particular, after pre-training, all parameters in the fast and slow networks are retained except for the last M fully connected layers; During scenario-specific training, the last M fully connected layers in the fast and slow networks are trained during the first N training iterations. During the remaining training iterations, the last M fully connected layers and layers M-1 to MT are trained. The learning rate of layers M-1 to MT is less than that of the last M layers.

[0047] In one or more alternative embodiments, M is 1 and T is 2.

[0048] The learning rates of layers M-1 to MT and the last M layer can be pre-configured manually. For example, the learning rate of layers M-1 to MT can be configured to be 0.0001, and the learning rate of the last M layer can be configured to be 0.001.

[0049] In other words, during pre-training, the entire training set is used for training. After training, all parameters except the last fully connected (FC) layer are retained as a highly generalizable pre-trained model. When used in a specific application scenario, the corresponding abnormal behavior videos that need to be identified within that scenario are extracted and trained on the pre-trained model. In the first 10 training iterations, only the last FC layer is trained, and in the next 10 iterations, the learning rate of the last FC layer is 0.001. The learning rate of the first two FC layers in the prediction module is 0.0001, thus obtaining a specialized video abnormal behavior recognition model for the specific scenario.

[0050] As an example, give as follows Figure 3 The diagram shows a prediction and identification module for a fast / slow network. Figure 3 As shown, the prediction and recognition module includes three fully connected layers, namely fc1, fc2, and fc3. fc3 comprises two layers, fc3-1 and fc3-2. During pre-training, all parameters except for the last layer fc3 are retained, while the parameters of the other layers are discarded, resulting in a pre-trained model. When used for specific scenarios, video data of specific abnormal behaviors is used to train the pre-trained model. The first 10 training iterations train only the fc3 layer, and the learning rate for fc3 in the subsequent 10 iterations is 0.001. The learning rates for fc1 and fc2 in the prediction module are also 0.0001. The final model obtained is one that can accurately detect specific abnormal behaviors.

[0051] In this embodiment, by introducing a multi-branch prediction network structure and training algorithm, the generalization performance of the model before the last fully connected (FC) layer is improved, and the specialization of the last FC layer is enhanced, thus training a model with more specialized recognition capabilities for various abnormal behaviors.

[0052] In related technologies, most video abnormal behavior recognition methods use simple three-layer fully connected recognition networks to obtain a single video abnormal behavior recognition model. This model is used for different abnormal behaviors, resulting in insufficient specialization. Furthermore, when new abnormal behavior categories are introduced, it is difficult to achieve excellent recognition results through simple training. This application proposes a multi-branch prediction network structure and training method to improve the generalization of the model before the last fully connected layer. For different abnormal behaviors, the last fully connected layer is trained specifically to obtain a specialized model, thereby improving the abnormal behavior recognition rate in real-world application scenarios.

[0053] The Fastslow network uses two paths: a fast pathway for capturing rapidly changing actions with a fast refresh rate and high temporal resolution, and a slow pathway for capturing semantic information provided by images and sparse frames.

[0054] The information connection methods typically employ three types of approaches: direct connection with dimensional transformation, time sampling, and time 3D convolution. These methods process the fast channel features to achieve the same time and spatial dimensions as the slow channel features, and then directly connect them to the slow channel features in the channel dimension.

[0055] While these three connection methods allow information exchange between the fast and slow channels, they have two problems. First, all three methods simply input information from the fast channel to the slow channel in a one-way manner, failing to fully utilize the advantages of both channels. Second, these three methods merely transform the feature dimensions of the fast channel to match those of the slow channel before performing a simple channel connection, without effectively considering the differences between the two feature sets.

[0056] To address the two issues mentioned above, this application improves feature effectiveness by designing a fast and slow information fusion system that bidirectionally interacts the information corresponding to the slow and fast channels.

[0057] The following section describes the fast and slow channel information fusion method of this embodiment, taking the first channel feature as a fast channel feature and the second channel feature as a slow channel feature as an example. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of fast and slow channel information fusion provided in an embodiment of this application.

[0058] like Figure 4 As shown, let the fast channel characteristics be... Common direct frame extraction operations can lead to the loss of inter-frame information, so a convolutional kernel is used here. Output Channel Step size is 3D convolution reduces inter-frame information loss, and the resulting dimension is... The features are then processed using max and average operations to obtain two values. Then cascade acquisition Location information features are obtained by performing a 2D convolution operation with a sigmoid function to obtain a time frame T. Location attention map infographic, and finally feature map and slow features Location information map is obtained by splitting T-frames. and feature map According to the formula Obtain the features of the corresponding T frames, and then stitch them back together. feature Afterwards, it is combined with the original slow characteristic. The summation yields the final new slow channel feature.

[0059] The following section describes the fast and slow channel information fusion method of this embodiment, taking the first channel feature as a slow channel feature and the second channel feature as a fast channel feature as an example. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of fast and slow channel information fusion provided in an embodiment of this application.

[0060] like Figure 5 As shown, let the slow channel feature be... To augment features to a dimension of 3 through frame augmentation After the features are identified, they are then processed separately. The maxpooling and avgpooling operations obtain two... Then cascade acquisition Channel information features are generated through a fully connected, FC layer with sigmoid function. Channel information map, and finally feature map and fast features Channel information map is obtained by splitting the T-frame. and feature map According to the formula Obtain the features of the corresponding T frames, and then stitch them back together. feature Then, compared with the original fast features The summation yields the final new fast channel features.

[0061] Most existing video abnormal behavior recognition methods use the fast-slow network framework as a baseline. They use fast and slow channel features with different frame sampling frequencies, and import the fast channel features into the slow channel features through a selected connection method. However, they do not introduce the slow channel feature information into the fast channel, and the connection method does not take into account the differences between the two channel features. This makes it difficult to effectively fuse the slow and fast channel information and to give full play to the effectiveness of the two features.

[0062] This invention enables bidirectional interaction between two channels through a fast and slow information fusion section, transforming the features of one channel into information that the other channel can understand. This improves the effectiveness of feature information, reduces information redundancy, and allows features to more accurately express video characteristics, thereby improving the accuracy of abnormal behavior recognition.

[0063] In one or more alternative embodiments, the loss function uses the binary cross-entropy loss commonly used in this field. Since anomalous behavior typically constitutes a small percentage of video frames, data augmentation is necessary to increase the number of positive samples.

[0064] This disclosure also provides a fast / slow channel information fusion apparatus for performing the fast / slow channel information fusion method provided in any of the above embodiments. Figure 6 As shown, the device includes: The acquisition module 61 is used to acquire the first channel feature and the second channel feature in the fast and slow network. The first channel feature is the feature output by the convolutional layer of the first channel in the fast and slow network. The second channel feature is the feature output by the convolutional layer corresponding to the convolutional layer in the second channel of the fast and slow network. The first channel and the second channel are different. The first channel is either a fast channel or a slow channel, and the second channel is either the fast channel or the slow channel. The dimensionality reduction or frame expansion module 62 is used to reduce the dimensionality or expand the frame of the first channel feature so that the frame number dimension of the first channel feature and the second channel feature are the same, thereby obtaining the first temporary channel feature. Pooling module 63 is used to perform pooling operation on the first temporary channel feature to obtain the second temporary channel feature; Generation module 64 is used to generate a feature map based on the features of the second temporary channel; The module 65 is used to obtain a new first channel feature based on the feature map, the first channel feature, and the second channel feature; Processing module 66 is used to process the new first channel feature using the first channel.

[0065] In one or more alternative embodiments, the pooling module 63 is used for: The first temporary channel feature is subjected to average pooling to obtain average pooled feature; and the first temporary channel feature is subjected to max pooling to obtain max pooled feature. The average pooling feature and the max pooling feature are concatenated to obtain the second temporary channel feature.

[0066] In one or more alternative embodiments, generation module 64 is used to: When the first channel is a fast channel, the features of the second temporary channel are subjected to 2D convolution processing to obtain the feature map; When the first channel is a slow channel, the second temporary channel features are fully connected to obtain the feature map.

[0067] In one or more alternative embodiments, the obtaining module 65 is used for: Obtain the number of video frames processed in one pass of the slow channel; For the given number of frames, the feature map is split to obtain a split feature map; Based on the feature map of the second channel features and the split feature map, the features of the video frame are obtained; The features of the video frame are added to the features of the first channel to obtain the new features of the first channel.

[0068] In one or more alternative embodiments, the training process of the fast and slow networks includes pre-training and scenario-specific training; In this process, after the pre-training is completed, all parameters in the fast and slow networks are retained except for the last M fully connected layers. During the scenario-specific training process, in the first N training iterations, the last M fully connected layers in the fast and slow network are trained. In the remaining training iterations, the last M fully connected layers and layers M-1 to MT are trained, and the learning rate of layers M-1 to MT is less than the learning rate of the last M layers.

[0069] In one or more alternative embodiments, M is 1 and T is 2.

[0070] The fast and slow channel information fusion device and the fast and slow channel information fusion method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0071] This disclosure also provides an electronic device for performing the above-described fast and slow channel information fusion method. Please refer to... Figure 7 It illustrates a schematic diagram of an electronic device provided by some embodiments of this disclosure. For example... Figure 7 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702, and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 702. The memory 701 stores a computer program that can run on the processor 700. When the processor 700 runs the computer program, it executes the fast and slow channel information fusion method provided in any of the foregoing embodiments of this disclosure.

[0072] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between the device network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0073] Bus 702 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 701 is used to store programs. After receiving an execution instruction, the processor 700 executes the program. The fast and slow channel information fusion method disclosed in any of the foregoing embodiments of this disclosure can be applied to the processor 700, or implemented by the processor 700.

[0074] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the steps of the above method.

[0075] The electronic device provided in this disclosure and the fast / slow channel information fusion method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0076] This disclosure also provides a computer-readable storage medium corresponding to the fast and slow channel information fusion method provided in the foregoing embodiments. The computer-readable storage medium is an optical disc, on which a computer program (i.e., a computer program product) is stored. When the computer program is run by a processor, it executes the fast and slow channel information fusion method provided in any of the foregoing embodiments.

[0077] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0078] The computer-readable storage medium provided in the above embodiments of this disclosure and the fast and slow channel information fusion method provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0079] This disclosure also provides a computer program product; please refer to [link / reference]. Figure 8 The computer program product 800 carries program code, namely computer program 801. The instructions included in the computer program 801 can be used to execute the steps of the fast and slow channel information fusion method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0080] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0081] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0082] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0083] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0084] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0085] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0086] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0087] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for fusing fast and slow channel information, characterized in that, include: Obtain the first channel feature and the second channel feature in the fast and slow network. The first channel feature is the feature output by the convolutional layer of the first channel in the fast and slow network. The second channel feature is the feature output by the convolutional layer corresponding to the convolutional layer in the second channel of the fast and slow network. The first channel and the second channel are different. The first channel is either the fast channel or the slow channel, and the second channel is either the fast channel or the slow channel. The first channel feature is reduced in dimension or expanded in frame so that the first channel feature and the second channel feature have the same frame number dimension, thus obtaining the first temporary channel feature; The first temporary channel feature is pooled to obtain the second temporary channel feature; A feature map is generated based on the features of the second temporary channel; Based on the feature map, the first channel feature, and the second channel feature, a new first channel feature is obtained; The new first channel features are processed using the first channel.

2. The method according to claim 1, characterized in that, The first temporary channel feature is pooled to obtain the second temporary channel feature, including: The first temporary channel feature is subjected to average pooling to obtain average pooled feature; and the first temporary channel feature is subjected to max pooling to obtain max pooled feature. The average pooling feature and the max pooling feature are concatenated to obtain the second temporary channel feature.

3. The method according to claim 1, characterized in that, A feature map is generated based on the features of the second temporary channel, including: When the first channel is a fast channel, the features of the second temporary channel are subjected to 2D convolution processing to obtain the feature map; When the first channel is a slow channel, the second temporary channel features are fully connected to obtain the feature map.

4. The method according to claim 1, characterized in that, Based on the feature map, the first channel feature, and the second channel feature, a new first channel feature is obtained, including: Obtain the number of video frames processed in one pass of the slow channel; For the given number of frames, the feature map is split to obtain a split feature map; Based on the feature map of the second channel features and the split feature map, the features of the video frame are obtained; The features of the video frame are added to the features of the first channel to obtain the new features of the first channel.

5. The method according to claim 1, characterized in that, The training process of the fast and slow networks includes pre-training and scenario-specific training; In this process, after the pre-training is completed, all parameters in the fast and slow networks are retained except for the last M fully connected layers. During the scenario-specific training process, in the first N training iterations, the last M fully connected layers in the fast and slow network are trained. In the remaining training iterations, the last M fully connected layers and layers M-1 to MT are trained, and the learning rate of layers M-1 to MT is less than the learning rate of the last M layers.

6. The method according to claim 5, characterized in that, M is 1, and T is 2.

7. A fast and slow channel information fusion device, characterized in that, include: The acquisition module is used to acquire the first channel features and the second channel features in the fast and slow network. The first channel features are the features output by the convolutional layer of the first channel in the fast and slow network, and the second channel features are the features output by the convolutional layer corresponding to the convolutional layer in the second channel of the fast and slow network. The first channel and the second channel are different. The first channel is either the fast channel or the slow channel, and the second channel is either the fast channel or the slow channel. A dimensionality reduction or frame expansion module is used to reduce the dimensionality or expand the frame of the first channel feature so that the frame number dimension of the first channel feature and the second channel feature are the same, thereby obtaining a first temporary channel feature. A pooling module is used to perform a pooling operation on the first temporary channel feature to obtain the second temporary channel feature; The generation module is used to generate a feature map based on the features of the second temporary channel; The acquisition module is used to obtain a new first channel feature based on the feature map, the first channel feature, and the second channel feature; The processing module is used to process the new first channel feature using the first channel.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

Images