Three-dimensional convolutional network model processing method and device, and non-volatile storage medium

Abstract

The present application discloses a method and device for processing a three-dimensional convolutional network model, and a non-volatile storage medium. The method includes: obtaining an original three-dimensional convolutional network model; performing decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, and the temporal convolution and the spatial convolution respectively correspond to different redundancies; performing compression processing on the temporal convolution and the spatial convolution according to the redundancies respectively corresponding to the temporal convolution and the spatial convolution; and outputting the compression result. The present application solves the technical problem in the prior art that the 3D convolutional network model reduces the operating performance and computing speed when performing 3D convolution operations.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for processing a three-dimensional convolutional network model, and a non-volatile storage medium. Background Art

[0002] In related technologies, 3D convolutional network models are widely used in video analysis and understanding tasks. Among them, 3D convolution operations are an important part of 3D convolutional network models. Due to the large number of calculation parameters and high computational complexity of 3D convolution operations, the 3D convolutional network models in existing technologies have reduced performance and speed when performing 3D convolution operations, which seriously affects the learning cycle of video training tasks.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for processing a three-dimensional convolutional network model, and a non-volatile storage medium, so as to at least solve the technical problem in the prior art that the 3D convolutional network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0005] According to one aspect of an embodiment of the present application, a method for processing a three-dimensional convolutional network model is provided, including: obtaining an original three-dimensional convolutional network model; performing decoupling processing on the above-mentioned original three-dimensional convolutional network model to obtain a decoupling result, wherein the above-mentioned decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the above-mentioned temporal convolution and the above-mentioned spatial convolution correspond to different redundancies respectively; compressing the above-mentioned temporal convolution and the above-mentioned spatial convolution according to the redundancies corresponding to the above-mentioned temporal convolution and the above-mentioned spatial convolution respectively; and outputting the compressed result.

[0006] According to another aspect of an embodiment of the present application, a processing device for a three-dimensional convolutional network model is also provided, including: an acquisition module for acquiring an original three-dimensional convolutional network model; a decoupling module for performing decoupling processing on the above-mentioned original three-dimensional convolutional network model to obtain a decoupling result, wherein the above-mentioned decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the above-mentioned temporal convolution and the above-mentioned spatial convolution correspond to different redundancies respectively; a processing module for compressing the above-mentioned temporal convolution and the above-mentioned spatial convolution according to the redundancies corresponding to the above-mentioned temporal convolution and the above-mentioned spatial convolution respectively; and an output module for outputting the compression result.

[0007] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned three-dimensional convolutional network model processing methods.

[0008] According to another aspect of an embodiment of the present application, a processing device for a three-dimensional convolutional network model is also provided, including: a processor; and a memory, connected to the above-mentioned processor, for providing the above-mentioned processor with instructions for processing the following processing steps: obtaining the original three-dimensional convolutional network model; performing decoupling processing on the above-mentioned original three-dimensional convolutional network model to obtain a decoupling result, wherein the above-mentioned decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the above-mentioned temporal convolution and the above-mentioned spatial convolution correspond to different redundancies respectively; compressing the above-mentioned temporal convolution and the above-mentioned spatial convolution according to the redundancies corresponding to the above-mentioned temporal convolution and the above-mentioned spatial convolution respectively; and outputting the compressed result.

[0009] According to another aspect of an embodiment of the present application, a method for processing a three-dimensional convolutional network model is also provided, including: a client receives a training instruction and obtains a sample video set, wherein the sample video set is a live video collected within a historical time period, and the sample videos in the sample video set output target objects by playing video images; the client executes the training of the sample video set using a three-dimensional convolutional network model based on the training instruction to obtain a target model, wherein the three-dimensional convolutional network model is a compression result after compression processing is performed on the original three-dimensional convolutional network model; wherein the compression processing is performed on the original three-dimensional convolutional network model, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compression result.

[0010] According to another aspect of an embodiment of the present application, a method for processing a three-dimensional convolutional network model is also provided, including: obtaining a video played during a live broadcast, the video including: multiple frames of video images; using a target network model to analyze each frame of the video image, and identifying the target object in the video image, wherein the target network model is generated by compressing the original three-dimensional convolutional network model; wherein the compression processing is performed on the original three-dimensional convolutional network model, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compression result.

[0011] According to another aspect of an embodiment of the present application, a method for processing a three-dimensional convolutional network model is also provided, including: a cloud server receives a model compression request, wherein the model compression request includes: identification information of the convolutional model that needs to be compressed; the cloud server retrieves the original three-dimensional convolutional network model that needs to be compressed based on the compression request; the cloud server decouples the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; the cloud server compresses the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively to generate a compression result; the cloud server feeds back the compression result to the front-end device.

[0012] In an embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to obtain an original three-dimensional convolutional network model; the original three-dimensional convolutional network model is decoupled to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively, the temporal convolution and the spatial convolution are compressed; and the compressed result is output.

[0013] It is easy to notice that the embodiment of the present application decouples the above-mentioned original three-dimensional convolutional network model to obtain temporal convolution, spatial convolution and feature reuse convolution, and since temporal convolution and spatial convolution correspond to different redundancies respectively; according to the different redundancies of temporal convolution and spatial convolution, different degrees of compression are given to spatial convolution and temporal convolution, so that the calculation speed is significantly improved.

[0014] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a processing method for a three-dimensional convolutional network model is shown;

[0017] Figure 2 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application;

[0018] Figure 3 This is a processing flow chart of an optional three-dimensional convolutional network model according to an embodiment of the present application;

[0019] Figure 4 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application;

[0020] Figure 5 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application;

[0021] Figure 6 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application;

[0022] Figure 7 is a structural diagram of a processing device for a three-dimensional convolutional network model according to an embodiment of the present application;

[0023] Figure 8 is a structural diagram of a processing device for a three-dimensional convolutional network model according to an embodiment of the present application;

[0024] Figure 9 It is a structural block diagram of another computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0028] 3D Convolution: refers to the three-dimensional convolution operation of space and time in the convolutional neural network model.

[0029] DeST3d: Decoupled Spatio-Temporal 3D convolution refers to the fast spatiotemporal decoupled three-dimensional convolution operation method proposed in this application.

[0030] Example 1

[0031] According to an embodiment of the present application, an embodiment of a method for processing a three-dimensional convolutional network model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] The method embodiment provided in Example 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a processing method for a three-dimensional convolutional network model is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the processing method of the three-dimensional convolutional network model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned processing method of the three-dimensional convolutional network model. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0035] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0036] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0037] Under the above operating environment, this application provides Figure 2 A processing method for a three-dimensional convolutional network model is shown. Figure 2 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application, such as Figure 2 As shown, the processing method of the above three-dimensional convolutional network model includes the following steps:

[0038] Step S202, obtaining the original three-dimensional convolutional network model;

[0039] Step S204: Decoupling the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively;

[0040] Step S206: compressing the temporal convolution and the spatial convolution according to their respective redundancies.

[0041] Step S208: output the compression result.

[0042] In an embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to obtain an original three-dimensional convolutional network model; the original three-dimensional convolutional network model is decoupled to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively, the temporal convolution and the spatial convolution are compressed; and the compressed result is output.

[0043] It is easy to notice that the embodiment of the present application decouples the above-mentioned original three-dimensional convolutional network model to obtain temporal convolution, spatial convolution and feature reuse convolution, and since temporal convolution and spatial convolution correspond to different redundancies respectively; according to the different redundancies of temporal convolution and spatial convolution, different degrees of compression are given to spatial convolution and temporal convolution, so that the calculation speed is significantly improved.

[0044] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0045] It should be noted that the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application can be applied to, but is not limited to, video understanding and its related application scenarios, video analysis and its related application scenarios, video processing and its related application scenarios, such as product recognition in live video, human body tracking and recognition, etc.

[0046] Based on the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to seamlessly upgrade the existing computer vision SOTA three-dimensional convolutional network model (i.e., the original three-dimensional convolutional network model), thereby significantly accelerating its training speed without affecting the accuracy performance of the three-dimensional convolutional network model.

[0047] Since the three-dimensional convolution operation is the basic operation in the three-dimensional convolutional network model, by replacing the three-dimensional convolution operation with the three-dimensional convolution decoupling operation, the amount of calculation parameters and the complexity of the calculation can be significantly reduced. Optionally, the existing decoupling formula can be used to decouple the original three-dimensional convolutional network model, or the Pytorch tool can be used to decouple the original three-dimensional convolutional network model to obtain a decoupling result.

[0048] In an embodiment of the present application, the temporal convolution and the spatial convolution correspond to different redundancies. For example, the temporal convolution corresponds to a first redundancy, and the spatial convolution corresponds to a second redundancy. Based on performance considerations, the second redundancy of the spatial convolution is greater, for example, 75%, while the first redundancy of the temporal convolution is less, for example, 25%. Based on the redundancies corresponding to the temporal convolution and the spatial convolution, the temporal convolution and the spatial convolution are compressed to different degrees to obtain and output a compressed result.

[0049] The solution of this application decouples the original computationally intensive three-dimensional convolutional network model into three branched operations, namely, a temporal convolution flow, a spatial convolution flow, and a feature reuse branch, thereby achieving lower computational complexity. Different degrees of compression are applied to the redundancy corresponding to the spatial convolution and temporal convolution, respectively, so that the processed three-dimensional convolutional network model has lower computational complexity without affecting the performance of the model.

[0050] In an optional embodiment, decoupling processing is performed on the original three-dimensional convolutional network model to obtain a decoupling result, including:

[0051] Step S302, obtaining the temporal feature data and spatial feature data of the original three-dimensional convolutional network model;

[0052] Step S304, determining feature reuse data of the original three-dimensional convolutional network model;

[0053] Step S306, based on the proportion of the above-mentioned time feature data, the above-mentioned spatial feature data and the feature reuse data in the above-mentioned original three-dimensional convolutional network model, decouple the above-mentioned original three-dimensional convolutional network model into the above-mentioned time convolution, the above-mentioned spatial convolution and the above-mentioned feature reuse convolution, wherein the proportion of the above-mentioned feature reuse data in the above-mentioned original three-dimensional convolutional network model is fixed.

[0054] Optionally, in the embodiments of the present application, a high-speed convolution operation method based on Pytorch tools or CUDNN can be used, but is not limited to, to obtain the temporal feature data and spatial feature data of the original three-dimensional convolutional network model, and to determine the feature reuse data of the above-mentioned original three-dimensional convolutional network model.

[0055] Based on the proportion of the above-mentioned time feature data, the above-mentioned spatial feature data and the feature reuse data in the above-mentioned original three-dimensional convolutional network model, as shown in FIG. Figure 3 As shown, the original three-dimensional convolutional network model X i Decoupled into spatial convolution X s , temporal convolution X t and feature reuse convolution X r .

[0056] Optionally, the feature reuse data may account for a fixed proportion of the original 3D convolutional network model, for example, approximately one-eighth. The spatial feature data may account for two-thirds of the original 3D convolutional network model, and the temporal feature data may account for one-third of the original 3D convolutional network model excluding the feature reuse data. For example, if the 3D convolution operation in the original 3D convolutional network model is a 3*3*3 convolution, it is decoupled into a 3*3 spatial convolution and a 3*1 temporal convolution.

[0057] In an optional embodiment, determining feature reuse data of the original three-dimensional convolutional network model includes:

[0058] Step S402, randomly selecting the feature reuse data from the original three-dimensional convolutional network model; or

[0059] Step S404: Use the head data and / or tail data of the original three-dimensional convolutional network model as the feature reuse data.

[0060] In an embodiment of the present application, in order to ensure the performance and robustness of the three-dimensional convolutional network model, feature reuse data can be randomly selected from the original three-dimensional convolutional network model; or a part of the feature data can be cut from the head and tail of the original three-dimensional convolutional network model as feature reuse data, and then the above-mentioned feature reuse convolution is obtained based on the proportion of the feature reuse data in the original three-dimensional convolutional network model.

[0061] In an optional embodiment, after obtaining the decoupling result, the method further includes:

[0062] Step S502, obtaining the original convolution channel of the original three-dimensional convolutional network model;

[0063] Step S504: Split the original convolution channel according to the decoupling result to obtain a temporal convolution channel, a spatial convolution channel, and a feature reuse channel.

[0064] In an embodiment of the present application, since the proportion of feature reuse data in the above-mentioned original three-dimensional convolutional network model is fixed, the proportion of feature reuse channels in the original convolutional channels is fixed and is set corresponding to the proportion of feature reuse data in the original three-dimensional convolutional network model.

[0065] As an optional embodiment, the sizes of the temporal convolution channel and the spatial convolution channel split from the original convolution channel of the original three-dimensional convolution network model can be determined based on the size of the redundancy corresponding to the temporal convolution and the spatial convolution. For example, since the second redundancy of the spatial convolution is larger, for example, 75%; the second redundancy of the temporal convolution is smaller, for example, 25%; the channel of the spatial convolution channel is larger, and the channel of the temporal convolution channel is larger.

[0066] In an optional embodiment, after obtaining the temporal convolution channel, the spatial convolution channel, and the feature reuse channel, the method further includes:

[0067] Step S602, using the temporal convolution channel to transmit the temporal convolution;

[0068] Step S604: using the spatial convolution channel to transmit the spatial convolution;

[0069] Step S606: Use the feature reuse channel to transmit the feature reuse convolution.

[0070] Still Figure 3 As shown, the time convolution channel T is used to transmit the time convolution X t , use the spatial convolution channel S to transmit the above spatial convolution X s , and use feature reuse channel R to transmit feature reuse convolution X r .

[0071] As an optional embodiment, the embodiment of the present application adopts a parallel mechanism when transmitting spatial convolution, temporal convolution and feature reuse convolution, that is, by splitting the original convolution channel into a temporal convolution channel, a spatial convolution channel and a feature reuse channel, some convolution channels (such as spatial convolution channels) perform a 3*3 convolution operation, and some convolution channels (such as temporal convolution channels) perform a 3*1 convolution operation, and finally splicing them together, as shown in FIG. Figure 3 As shown, the original convolution channel is split before performing the convolution operation (i.e. Figure 3 The Channel Split process shown in the figure is divided into time convolution channel T, spatial convolution channel S and feature reuse channel R, and then convolution operation is performed separately, and then amplified and spliced ​​after the operation is completed (for example, you can use Figure 3 The concat splicing method shown in the figure) is used to obtain

[0072] In an optional embodiment, compressing the temporal convolution and the spatial convolution according to the redundancy corresponding to each of the temporal convolution and the spatial convolution includes:

[0073] Step S702: determining a first redundancy corresponding to the temporal convolution and a second redundancy corresponding to the spatial convolution, wherein the second redundancy is greater than the first redundancy;

[0074] Step S704: determining a first compression level corresponding to the temporal convolution and a second compression level corresponding to the spatial convolution based on the first redundancy and the second redundancy;

[0075] Step S706 , performing a first compression process on the temporal convolution according to the first compression level, and performing a second compression process on the spatial convolution according to the second compression level.

[0076] In an embodiment of the present application, the temporal convolution and the spatial convolution correspond to different redundancies, respectively. For example, the temporal convolution corresponds to a first redundancy, and the spatial convolution corresponds to a second redundancy. Based on performance considerations, the second redundancy of the spatial convolution is greater, for example, 75%, while the first redundancy of the temporal convolution is smaller, for example, 25%. Based on the magnitude of the first and second redundancies, a first compression level corresponding to the temporal convolution and a second compression level corresponding to the spatial convolution are determined. The temporal convolution is then subjected to a first compression process based on the first compression level, and the spatial convolution is subjected to a second compression process based on the second compression level.

[0077] In an optional embodiment, the compression result includes: the first compression result of the temporal convolution and the second compression result of the spatial convolution; after outputting the compression result, the method further includes:

[0078] Step S802: Randomly sort the first compression result, the second compression result, and the feature reuse convolution according to a predetermined rule to obtain a random sorting result;

[0079] Step S804: outputting the first compression result, the second compression result, and the feature reuse convolution based on the random sorting result.

[0080] Step S804: concatenate the first compression result, the second compression result, and the feature reuse convolution to obtain a processed three-dimensional network model.

[0081] Still Figure 3 As shown, the random sorting model Shuffle is used to compress the first compression result according to the predetermined rules. The second compression result is And the above feature reuse convolution is Perform random sorting and obtain random sorting result X o , based on the above random sorting result X o , output the above-mentioned first compression result, the above-mentioned second compression result and the above-mentioned feature reuse convolution, and obtain the processed three-dimensional network model by splicing the above-mentioned first compression result, the above-mentioned second compression result and the above-mentioned feature reuse convolution.

[0082] Under the above operating environment, this application provides Figure 4 Another processing method for the three-dimensional convolutional network model is shown. Figure 4 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application, such as Figure 4 As shown, the processing method of the above three-dimensional convolutional network model includes the following steps:

[0083] Step S902: The client receives the training instruction and obtains a sample video set, wherein the sample video set is a live video collected within a historical time period, and the sample videos in the sample video set output the target object by playing the video screen;

[0084] In step S904, the client executes the training of the sample video set using the three-dimensional convolutional network model based on the training instruction to obtain a target model, wherein the three-dimensional convolutional network model is a compressed result after the original three-dimensional convolutional network model is compressed; wherein the compression processing is performed on the original three-dimensional convolutional network model, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compression result.

[0085] In an embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted, a training instruction is received through the client, and a sample video set is obtained, wherein the above-mentioned sample video set is a live video collected within a historical time period, and the sample videos in the above-mentioned sample video set output the target object by playing the video screen; the client executes the above-mentioned sample video set based on the above-mentioned training instruction using a three-dimensional convolutional network model to train the above-mentioned sample video set to obtain a target model, wherein the above-mentioned three-dimensional convolutional network model is a compression result after compression processing is performed on the original three-dimensional convolutional network model; wherein the above-mentioned compression processing is performed on the above-mentioned original three-dimensional convolutional network model, including: decoupling the above-mentioned original three-dimensional convolutional network model to obtain a decoupling result, and compressing the above-mentioned decoupling result according to the redundancy corresponding to the above-mentioned decoupling result, and outputting the above-mentioned compression result.

[0086] It is easy to notice that the client of the embodiment of the present application receives a training instruction and obtains a sample video set. By decoupling the above-mentioned original three-dimensional convolutional network model, temporal convolution, spatial convolution and feature reuse convolution are obtained. Moreover, since temporal convolution and spatial convolution correspond to different redundancies respectively, according to the different redundancies of temporal convolution and spatial convolution, different degrees of compression are given to spatial convolution and temporal convolution, which significantly improves the calculation speed.

[0087] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0088] It should be noted that the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application can be applied to, but is not limited to, video understanding and its related application scenarios, video analysis and its related application scenarios, video processing and its related application scenarios, such as product recognition in live video, human body tracking and recognition, etc.

[0089] Based on the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to seamlessly upgrade the existing computer vision SOTA three-dimensional convolutional network model (i.e., the original three-dimensional convolutional network model), thereby significantly accelerating its training speed without affecting the accuracy performance of the three-dimensional convolutional network model.

[0090] Since the three-dimensional convolution operation is the basic operation in the three-dimensional convolutional network model, by replacing the three-dimensional convolution operation with the three-dimensional convolution decoupling operation, the amount of calculation parameters and the complexity of the calculation can be significantly reduced. Optionally, the existing decoupling formula can be used to decouple the original three-dimensional convolutional network model, or the Pytorch tool can be used to decouple the original three-dimensional convolutional network model to obtain a decoupling result.

[0091] In an embodiment of the present application, the temporal convolution and the spatial convolution correspond to different redundancies. For example, the temporal convolution corresponds to a first redundancy, and the spatial convolution corresponds to a second redundancy. Based on performance considerations, the second redundancy of the spatial convolution is greater, for example, 75%, while the first redundancy of the temporal convolution is less, for example, 25%. Based on the redundancies corresponding to the temporal convolution and the spatial convolution, the temporal convolution and the spatial convolution are compressed to different degrees to obtain and output a compressed result.

[0092] The solution of this application decouples the original computationally intensive three-dimensional convolutional network model into three branched operations, namely, a temporal convolution flow, a spatial convolution flow, and a feature reuse branch, thereby achieving lower computational complexity. Different degrees of compression are applied to the redundancy corresponding to the spatial convolution and temporal convolution, respectively, so that the processed three-dimensional convolutional network model has lower computational complexity without affecting the performance of the model.

[0093] Under the above operating environment, this application provides Figure 5 Another processing method for the three-dimensional convolutional network model is shown. Figure 5 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application, such as Figure 5 As shown, the processing method of the above three-dimensional convolutional network model includes the following steps:

[0094] Step S1002: obtaining a video played during a live broadcast, wherein the video includes: multiple frames of video images;

[0095] Step S1004: Analyze each video frame using the target network model to identify the target object in the video frame, wherein the target network model is generated by compressing the original three-dimensional convolutional network model;

[0096] Among them, performing the above-mentioned compression processing on the above-mentioned original three-dimensional convolutional network model includes: performing decoupling processing on the above-mentioned original three-dimensional convolutional network model to obtain a decoupling result, and compressing the above-mentioned decoupling result according to the redundancy corresponding to the above-mentioned decoupling result, and outputting the above-mentioned compression result.

[0097] In an embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted, by obtaining a video played during a live broadcast, the video including: multiple frames of video images; using a target network model to analyze each frame of the video image, and identifying the target object in the video image, wherein the target network model is generated by compressing the original three-dimensional convolutional network model; wherein the compression processing is performed on the original three-dimensional convolutional network model, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compression result.

[0098] It is easy to notice that the embodiment of the present application obtains the video played during the live broadcast, uses the target network model to analyze each frame of the video, identifies the target object in the above video, and decouples the above original three-dimensional convolutional network model to obtain time convolution, spatial convolution and feature reuse convolution, and because time convolution and spatial convolution correspond to different redundancies respectively; according to the different redundancies of time convolution and spatial convolution, different degrees of compression are given to spatial convolution and time convolution, so that the calculation speed is significantly improved.

[0099] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0100] It should be noted that the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application can be applied to, but is not limited to, video understanding and its related application scenarios, video analysis and its related application scenarios, video processing and its related application scenarios, such as product recognition in live video, human body tracking and recognition, etc.

[0101] Based on the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to seamlessly upgrade the existing computer vision SOTA three-dimensional convolutional network model (i.e., the original three-dimensional convolutional network model), thereby significantly accelerating its training speed without affecting the accuracy performance of the three-dimensional convolutional network model.

[0102] Since the three-dimensional convolution operation is the basic operation in the three-dimensional convolutional network model, by replacing the three-dimensional convolution operation with the three-dimensional convolution decoupling operation, the amount of calculation parameters and the complexity of the calculation can be significantly reduced. Optionally, the existing decoupling formula can be used to decouple the original three-dimensional convolutional network model, or the Pytorch tool can be used to decouple the original three-dimensional convolutional network model to obtain a decoupling result.

[0103] In an embodiment of the present application, the temporal convolution and the spatial convolution correspond to different redundancies. For example, the temporal convolution corresponds to a first redundancy, and the spatial convolution corresponds to a second redundancy. Based on performance considerations, the second redundancy of the spatial convolution is greater, for example, 75%, while the first redundancy of the temporal convolution is less, for example, 25%. Based on the redundancies corresponding to the temporal convolution and the spatial convolution, the temporal convolution and the spatial convolution are compressed to different degrees to obtain and output a compressed result.

[0104] The solution of this application decouples the original computationally intensive three-dimensional convolutional network model into three branched operations, namely, a temporal convolution flow, a spatial convolution flow, and a feature reuse branch, thereby achieving lower computational complexity. Different degrees of compression are applied to the redundancy corresponding to the spatial convolution and temporal convolution, respectively, so that the processed three-dimensional convolutional network model has lower computational complexity without affecting the performance of the model.

[0105] Under the above operating environment, this application provides Figure 6 Another processing method for the three-dimensional convolutional network model is shown. Figure 6 is a flowchart of a method for processing a three-dimensional convolutional network model according to an embodiment of the present application, such as Figure 6 As shown, the processing method of the above three-dimensional convolutional network model includes the following steps:

[0106] Step S1102: The cloud server receives a model compression request, wherein the model compression request includes: identification information of the convolutional model that needs to be compressed;

[0107] Step S1104: The cloud server retrieves the original three-dimensional convolutional network model that needs to be compressed based on the compression request.

[0108] Step S1006: The cloud server performs decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively.

[0109] Step S1108: The cloud server compresses the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution, respectively, to generate a compression result.

[0110] In step S1110, the cloud server feeds back the compression result to the front-end device.

[0111] In an embodiment of the present application, a model compression request is received through a cloud server, wherein the model compression request includes: identification information of a convolutional model that needs to be compressed; the cloud server retrieves the original three-dimensional convolutional network model that needs to be compressed based on the compression request; the cloud server decouples the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies, respectively; the cloud server compresses the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution, respectively, to generate a compression result; the cloud server feeds back the compression result to the front-end device.

[0112] It is easy to notice that after the cloud server in the embodiment of the present application receives the model compression request, it retrieves the original three-dimensional convolutional network model that needs to perform the above-mentioned compression processing based on the above-mentioned compression request, and decouples the above-mentioned original three-dimensional convolutional network model to obtain time convolution, spatial convolution and feature reuse convolution, and because time convolution and spatial convolution correspond to different redundancies respectively; according to the different redundancies of time convolution and spatial convolution, the cloud server gives different degrees of compression to spatial convolution and time convolution, so that the calculation speed is significantly improved.

[0113] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0114] It should be noted that the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application can be applied to, but is not limited to, video understanding and its related application scenarios, video analysis and its related application scenarios, video processing and its related application scenarios, such as product recognition in live video, human body tracking and recognition, etc.

[0115] Based on the processing method embodiment of the three-dimensional convolutional network model provided in the embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to seamlessly upgrade the existing computer vision SOTA three-dimensional convolutional network model (i.e., the original three-dimensional convolutional network model), thereby significantly accelerating its training speed without affecting the accuracy performance of the three-dimensional convolutional network model.

[0116] Since the three-dimensional convolution operation is the basic operation in the three-dimensional convolutional network model, by replacing the three-dimensional convolution operation with the three-dimensional convolution decoupling operation, the amount of calculation parameters and the complexity of the calculation can be significantly reduced. Optionally, the existing decoupling formula can be used to decouple the original three-dimensional convolutional network model, or the Pytorch tool can be used to decouple the original three-dimensional convolutional network model to obtain a decoupling result.

[0117] In an embodiment of the present application, the temporal convolution and the spatial convolution correspond to different redundancies. For example, the temporal convolution corresponds to a first redundancy, and the spatial convolution corresponds to a second redundancy. Based on performance considerations, the second redundancy of the spatial convolution is greater, for example, 75%, while the first redundancy of the temporal convolution is less, for example, 25%. Based on the redundancies corresponding to the temporal convolution and the spatial convolution, the temporal convolution and the spatial convolution are compressed to different degrees to obtain and output a compressed result.

[0118] The solution of this application decouples the original computationally intensive three-dimensional convolutional network model into three branched operations, namely, a temporal convolution flow, a spatial convolution flow, and a feature reuse branch, thereby achieving lower computational complexity. Different degrees of compression are applied to the redundancy corresponding to the spatial convolution and temporal convolution, respectively, so that the processed three-dimensional convolutional network model has lower computational complexity without affecting the performance of the model.

[0119] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a non-volatile storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the above-mentioned methods of each embodiment of the present application.

[0121] Example 2

[0122] According to an embodiment of the present application, there is also provided an embodiment of a device for implementing the above-mentioned processing method of the three-dimensional convolutional network model. Figure 7 is a structural diagram of a processing device for a three-dimensional convolutional network model according to an embodiment of the present application, such as Figure 7 As shown, the device includes: an acquisition module 40, a decoupling module 42, a processing module 44 and an output module 46, wherein:

[0123] An acquisition module 40 is used to obtain the original three-dimensional convolutional network model; a decoupling module 42 is used to decouple the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; a processing module 44 is used to compress the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively; an output module 46 is used to output the compression result.

[0124] It should be noted that the acquisition module 40, decoupling module 42, processing module 44, and output module 46 correspond to steps S202 to S208 in Example 1. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0125] In an embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to obtain an original three-dimensional convolutional network model; the original three-dimensional convolutional network model is decoupled to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively, the temporal convolution and the spatial convolution are compressed; and the compressed result is output.

[0126] It is easy to notice that the embodiment of the present application decouples the above-mentioned original three-dimensional convolutional network model to obtain temporal convolution, spatial convolution and feature reuse convolution, and since temporal convolution and spatial convolution correspond to different redundancies respectively; according to the different redundancies of temporal convolution and spatial convolution, different degrees of compression are given to spatial convolution and temporal convolution, so that the calculation speed is significantly improved.

[0127] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0128] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0129] Example 3

[0130] According to an embodiment of the present application, an embodiment of a processing device for a three-dimensional convolutional network model is also provided. The processing device for the three-dimensional convolutional network model can be any computing device in a computing device group. Figure 8 is a structural diagram of a processing device for a three-dimensional convolutional network model according to an embodiment of the present application, such as Figure 8 As shown, the processing device of the three-dimensional convolutional network model includes: a processor 500 and a memory 502, wherein:

[0131] A processor 500; and a memory 502, connected to the above-mentioned processor 500, for providing the above-mentioned processor with instructions for processing the following processing steps: obtaining an original three-dimensional convolutional network model; performing decoupling processing on the above-mentioned original three-dimensional convolutional network model to obtain a decoupling result, wherein the above-mentioned decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the above-mentioned temporal convolution and the above-mentioned spatial convolution respectively correspond to different redundancies; compressing the above-mentioned temporal convolution and the above-mentioned spatial convolution according to the redundancies corresponding to the above-mentioned temporal convolution and the above-mentioned spatial convolution respectively; and outputting the compressed result.

[0132] In an embodiment of the present application, a fast three-dimensional convolution operation based on spatiotemporal decoupling is adopted to obtain an original three-dimensional convolutional network model; the original three-dimensional convolutional network model is decoupled to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively, the temporal convolution and the spatial convolution are compressed; and the compressed result is output.

[0133] It is easy to notice that the embodiment of the present application decouples the above-mentioned original three-dimensional convolutional network model to obtain temporal convolution, spatial convolution and feature reuse convolution, and since temporal convolution and spatial convolution correspond to different redundancies respectively; according to the different redundancies of temporal convolution and spatial convolution, different degrees of compression are given to spatial convolution and temporal convolution, so that the calculation speed is significantly improved.

[0134] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0135] It should be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0136] Example 4

[0137] According to an embodiment of the present application, a computer terminal embodiment is also provided, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0138] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0139] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the processing method of the three-dimensional convolutional network model of the application: obtaining the original three-dimensional convolutional network model; performing decoupling processing on the above-mentioned original three-dimensional convolutional network model to obtain a decoupling result, wherein the above-mentioned decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the above-mentioned temporal convolution and the above-mentioned spatial convolution correspond to different redundancies respectively; compressing the above-mentioned temporal convolution and the above-mentioned spatial convolution according to the redundancies corresponding to the above-mentioned temporal convolution and the above-mentioned spatial convolution respectively; and outputting the compressed result.

[0140] Optionally, Figure 9 is a structural block diagram of another computer terminal according to an embodiment of the present application, such as Figure 9 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 602 , a memory 604 , and a peripheral interface 606 .

[0141] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the processing method and device of the three-dimensional convolutional network model in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned processing method of the three-dimensional convolutional network model. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0142] Optionally, the processor may also execute the program code of the following steps: obtaining an original three-dimensional convolutional network model; performing decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; performing compression processing on the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively; and outputting the compression result.

[0143] Optionally, the processor may also execute the program code of the following steps: obtaining the temporal feature data and spatial feature data of the original three-dimensional convolutional network model; determining the feature reuse data of the original three-dimensional convolutional network model; and decoupling the original three-dimensional convolutional network model into the temporal convolution, the spatial convolution and the feature reuse convolution based on the proportion of the temporal feature data, the spatial feature data and the feature reuse data in the original three-dimensional convolutional network model, wherein the proportion of the feature reuse data in the original three-dimensional convolutional network model is fixed.

[0144] Optionally, the processor may also execute the program code of the following steps: randomly selecting the feature reuse data from the original three-dimensional convolutional network model; or using the header data and / or tail data of the original three-dimensional convolutional network model as the feature reuse data.

[0145] Optionally, the processor may also execute the program code of the following steps: obtaining the original convolution channel of the original three-dimensional convolutional network model; splitting the original convolution channel according to the decoupling result to obtain a temporal convolution channel, a spatial convolution channel and a feature reuse channel.

[0146] Optionally, the processor may further execute program code of the following steps: using the temporal convolution channel to transmit the temporal convolution; using the spatial convolution channel to transmit the spatial convolution; and using the feature reuse channel to transmit the feature reuse convolution.

[0147] Optionally, the processor may also execute the program code of the following steps: determining a first redundancy corresponding to the temporal convolution and a second redundancy corresponding to the spatial convolution, wherein the second redundancy is greater than the first redundancy; determining a first compression level corresponding to the temporal convolution and a second compression level corresponding to the spatial convolution based on the sizes of the first redundancy and the second redundancy; performing a first compression processing on the temporal convolution according to the first compression level, and performing a second compression processing on the spatial convolution according to the second compression level.

[0148] Optionally, the processor may also execute the program code of the following steps: randomly sorting the first compression result, the second compression result and the feature reuse convolution according to a predetermined rule to obtain a random sorting result; outputting the first compression result, the second compression result and the feature reuse convolution based on the random sorting result; and splicing the first compression result, the second compression result and the feature reuse convolution to obtain a processed three-dimensional network model.

[0149] An embodiment of the present application provides a processing scheme for a three-dimensional convolutional network model. The scheme comprises obtaining an original three-dimensional convolutional network model; performing decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, wherein the temporal convolution and spatial convolution respectively correspond to different redundancies; performing compression processing on the temporal convolution and spatial convolution according to the redundancies corresponding to the temporal convolution and spatial convolution, and outputting the compressed result.

[0150] It is easy to notice that the embodiment of the present application decouples the above-mentioned original three-dimensional convolutional network model to obtain temporal convolution, spatial convolution and feature reuse convolution, and since temporal convolution and spatial convolution correspond to different redundancies respectively; according to the different redundancies of temporal convolution and spatial convolution, different degrees of compression are given to spatial convolution and temporal convolution, so that the calculation speed is significantly improved.

[0151] Therefore, through the embodiments of the present application, the purpose of improving the computing speed and operating performance of the three-dimensional convolution network model in performing three-dimensional convolution operations is achieved, thereby achieving the technical effect of improving the model performance of the three-dimensional convolution network model as a whole, and further solving the technical problem in the prior art that the 3D convolution network model causes reduced operating performance and computing speed when performing 3D convolution operations.

[0152] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the processing method of the three-dimensional convolutional network model of the application: the client receives a training instruction and obtains a sample video set, wherein the sample video set is a live video collected within a historical time period, and the sample videos in the sample video set output the target object by playing the video screen; the client executes the training of the sample video set using the three-dimensional convolutional network model based on the training instruction to obtain the target model, wherein the three-dimensional convolutional network model is the compression result after the original three-dimensional convolutional network model is compressed; wherein, the original three-dimensional convolutional network model is compressed, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compressed result.

[0153] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the processing method of the three-dimensional convolutional network model of the application: obtaining the video played during the live broadcast, the video including: multiple frames of video images; using the target network model to analyze each frame of the video image, and identifying the target object in the video image, wherein the target network model is generated by compressing the original three-dimensional convolutional network model; wherein the original three-dimensional convolutional network model is compressed, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compressed result.

[0154] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the processing method of the three-dimensional convolutional network model of the application: the cloud server receives a model compression request, wherein the model compression request includes: identification information of the convolutional model that needs to be compressed; the cloud server retrieves the original three-dimensional convolutional network model that needs to be compressed based on the compression request; the cloud server decouples the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and spatial convolution correspond to different redundancies respectively; the cloud server compresses the temporal convolution and spatial convolution according to the redundancies corresponding to the temporal convolution and spatial convolution respectively, and generates a compression result; the cloud server feeds back the compression result to the front-end device.

[0155] It can be understood by those skilled in the art that Figure 9 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 9 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Figure 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 9 Different configurations shown.

[0156] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable non-volatile storage medium. The non-volatile storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0157] Example 5

[0158] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the method for processing a three-dimensional convolutional network model provided in the above embodiment.

[0159] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0160] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining an original three-dimensional convolutional network model; performing decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution correspond to different redundancies respectively; performing compression processing on the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively; and outputting the compression result.

[0161] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining the temporal feature data and spatial feature data of the above-mentioned original three-dimensional convolutional network model; determining the feature reuse data of the above-mentioned original three-dimensional convolutional network model; based on the proportion of the above-mentioned temporal feature data, the above-mentioned spatial feature data and the feature reuse data in the above-mentioned original three-dimensional convolutional network model, decoupling the above-mentioned original three-dimensional convolutional network model into the above-mentioned temporal convolution, the above-mentioned spatial convolution and the above-mentioned feature reuse convolution, wherein the proportion of the above-mentioned feature reuse data in the above-mentioned original three-dimensional convolutional network model is fixed.

[0162] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: randomly selecting the above-mentioned feature reuse data from the above-mentioned original three-dimensional convolutional network model; or using the header data and / or tail data of the above-mentioned original three-dimensional convolutional network model as the above-mentioned feature reuse data.

[0163] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining the original convolution channel of the above-mentioned original three-dimensional convolutional network model; splitting the above-mentioned original convolution channel according to the above-mentioned decoupling result to obtain a temporal convolution channel, a spatial convolution channel and a feature reuse channel.

[0164] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for performing the following steps: using the above-mentioned temporal convolution channel to transmit the above-mentioned temporal convolution; using the above-mentioned spatial convolution channel to transmit the above-mentioned spatial convolution; using the above-mentioned feature reuse channel to transmit the above-mentioned feature reuse convolution.

[0165] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining a first redundancy corresponding to the above-mentioned temporal convolution, and a second redundancy corresponding to the above-mentioned spatial convolution, wherein the above-mentioned second redundancy is greater than the above-mentioned first redundancy; based on the sizes of the above-mentioned first redundancy and the above-mentioned second redundancy, determining a first compression level corresponding to the above-mentioned temporal convolution and a second compression level corresponding to the above-mentioned spatial convolution; performing a first compression processing on the above-mentioned temporal convolution according to the above-mentioned first compression level, and performing a second compression processing on the above-mentioned spatial convolution according to the above-mentioned second compression level.

[0166] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: randomly sorting the above-mentioned first compression result, the above-mentioned second compression result and the above-mentioned feature reuse convolution according to a predetermined rule to obtain a random sorting result; based on the above-mentioned random sorting result, outputting the first compression result, the above-mentioned second compression result and the above-mentioned feature reuse convolution; splicing the above-mentioned first compression result, the above-mentioned second compression result and the above-mentioned feature reuse convolution to obtain a processed three-dimensional network model.

[0167] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: the client receives a training instruction and obtains a sample video set, wherein the sample video set is a live video collected within a historical time period, and the sample videos in the sample video set output the target object by playing the video screen; the client executes the training of the sample video set using a three-dimensional convolutional network model based on the training instruction to obtain a target model, wherein the three-dimensional convolutional network model is a compressed result after compression processing is performed on the original three-dimensional convolutional network model; wherein compression processing is performed on the original three-dimensional convolutional network model, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compressed result.

[0168] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining a video played during a live broadcast, the video including: multiple frames of video; using a target network model to analyze each frame of the video, and identifying a target object in the video, wherein the target network model is generated by compressing the original three-dimensional convolutional network model; wherein compression processing is performed on the original three-dimensional convolutional network model, including: decoupling the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compressed result.

[0169] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: the cloud server receives a model compression request, wherein the model compression request includes: identification information of the convolution model that needs to be compressed; the cloud server retrieves the original three-dimensional convolutional network model that needs to be compressed based on the compression request; the cloud server decouples the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and temporal convolution and spatial convolution correspond to different redundancies respectively; the cloud server compresses temporal convolution and spatial convolution according to the redundancies corresponding to temporal convolution and spatial convolution respectively, and generates a compression result; the cloud server feeds back the compression result to the front-end device.

[0170] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0171] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0173] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0175] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a non-volatile storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0176] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing a three-dimensional convolutional network model, characterized in that: include: Acquire video data, the video data including: multiple frames of video images; Analyze each frame of the video using a target network model to identify the target object in the video; The determination of the target network model includes: Acquire an original three-dimensional convolutional network model; based on the proportion of the temporal feature data, spatial feature data and feature reuse data of the original three-dimensional convolutional network model in the original three-dimensional convolutional network model, decouple the original three-dimensional convolutional network model into temporal convolution, spatial convolution and feature reuse convolution, the proportion of the feature reuse data in the original three-dimensional convolutional network model is fixed, and the temporal convolution and the spatial convolution respectively correspond to different redundancies; compress the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively; and output the compression result.

2. The method according to claim 1, characterized in that Determining feature reuse data of the original three-dimensional convolutional network model, including: The feature reuse data is randomly selected from the original three-dimensional convolutional network model.

3. The method according to claim 1, characterized in that Determining feature reuse data of the original three-dimensional convolutional network model, including: The head data and / or tail data of the original three-dimensional convolutional network model are used as the feature reuse data.

4. The method according to claim 1, wherein The method further comprises: Obtaining the original convolution channel of the original three-dimensional convolutional network model; According to the decoupling result, the original convolution channel is split using a parallel mechanism to obtain a temporal convolution channel, a spatial convolution channel and a feature reuse channel.

5. The method according to claim 4, characterized in that After obtaining the temporal convolution channel, the spatial convolution channel, and the feature reuse channel, the method further includes: Transmitting the temporal convolution using the temporal convolution channel; Transmitting the spatial convolution using the spatial convolution channel; The feature reuse convolution is transmitted using the feature reuse channel.

6. The method according to claim 1, characterized in that The method further comprises: performing compression processing on the temporal convolution and the spatial convolution according to redundancies corresponding to the temporal convolution and the spatial convolution, comprising: determining a first redundancy corresponding to the temporal convolution and a second redundancy corresponding to the spatial convolution, wherein the second redundancy is greater than the first redundancy; Determining, based on the magnitudes of the first redundancy and the second redundancy, a first compression level corresponding to the temporal convolution and a second compression level corresponding to the spatial convolution; A first compression process is performed on the temporal convolution according to the first compression level, and a second compression process is performed on the spatial convolution according to the second compression level.

7. The method according to any one of claims 1 to 6, characterized in that The compression result includes: a first compression result of the temporal convolution and a second compression result of the spatial convolution; after outputting the compression result, the method further includes: Randomly sorting the first compression result, the second compression result, and the feature reuse convolution according to a predetermined rule to obtain a random sorting result; Based on the random sorting result, outputting the first compression result, the second compression result and the feature reuse convolution; The first compression result, the second compression result, and the feature reuse convolution are spliced ​​to obtain a processed three-dimensional network model.

8. A processing device for a three-dimensional convolutional network model, characterized in that: include: The video processing module is used to obtain video data, wherein the video data includes: multiple frames of video images; use the target network model to analyze each frame of the video image and identify the target object in the video image; wherein the determination of the target network model includes: Acquisition module, used to obtain the original three-dimensional convolutional network model; A decoupling module, configured to perform decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, and the temporal convolution and the spatial convolution respectively correspond to different redundancies; a processing module, configured to perform compression processing on the temporal convolution and the spatial convolution according to redundancies corresponding to the temporal convolution and the spatial convolution respectively; Output module, used to output compression results.

9. A method for processing a three-dimensional convolutional network model, characterized in that: include: The client receives the training instruction and obtains a sample video set, wherein the sample video set is a live video collected within a historical time period, and the sample videos in the sample video set output the target object by playing the video screen; The client executes, based on the training instruction, training the sample video set using a three-dimensional convolutional network model to obtain a target model, wherein the three-dimensional convolutional network model is a compression result after performing compression processing on the original three-dimensional convolutional network model; Among them, performing the compression processing on the original three-dimensional convolutional network model includes: performing decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compressed result.

10. A method for processing a three-dimensional convolutional network model, characterized in that: include: Obtaining a video played during a live broadcast, the video including: multiple video frames; Analyzing each frame of the video using a target network model to identify a target object in the video, wherein the target network model is generated by compressing the original three-dimensional convolutional network model; Among them, performing the compression processing on the original three-dimensional convolutional network model includes: performing decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, and compressing the decoupling result according to the redundancy corresponding to the decoupling result, and outputting the compressed result.

11. A method for processing a three-dimensional convolutional network model, characterized in that: include: Acquire video data, the video data including: multiple frames of video images; Analyze each frame of the video using a target network model to identify the target object in the video; The determination of the target network model includes: The cloud server receives a model compression request, wherein the model compression request includes: identification information of a convolutional model that needs to be compressed; The cloud server retrieves the original three-dimensional convolutional network model that needs to perform the compression processing based on the compression request; The cloud server performs decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution, and feature reuse convolution, and the temporal convolution and the spatial convolution respectively correspond to different redundancies; The cloud server compresses the temporal convolution and the spatial convolution according to the redundancy corresponding to each of the temporal convolution and the spatial convolution to generate a compression result; The cloud server feeds back the compression result to the front-end device.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the processing method of the three-dimensional convolutional network model described in any one of claims 1 to 7.

13. A processing device for a three-dimensional convolutional network model, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Acquire video data, the video data including: multiple frames of video images; Analyze each frame of the video using a target network model to identify the target object in the video; The determination of the target network model includes: Acquire an original three-dimensional convolutional network model; perform decoupling processing on the original three-dimensional convolutional network model to obtain a decoupling result, wherein the decoupling result includes: temporal convolution, spatial convolution and feature reuse convolution, and the temporal convolution and the spatial convolution respectively correspond to different redundancies; compress the temporal convolution and the spatial convolution according to the redundancies corresponding to the temporal convolution and the spatial convolution respectively; and output the compression result.

Images