Deep network skipping method and apparatus based on similarity detection

Abstract

The application relates to the technical field of artificial intelligence, and provides a deep network skipping method and device based on similarity detection, which are applied to an encoder-decoder network. The deep network skipping method based on similarity detection comprises the following steps: current frame input data in a continuous time input sequence is acquired, and a shallow network in a previously delimited encoder-decoder network is used for feature extraction to obtain a current frame feature map; a frame of feature map recently saved in a feature cache unit is acquired, and the average similarity between the current frame feature map and the frame of feature map recently saved is calculated; the average similarity is compared with a preset similarity threshold to obtain a similarity judgment result; and whether a deep skipping instruction is triggered is dynamically decided according to the similarity judgment result. The application can dynamically skip the redundant calculation of the deep network corresponding to similar input under the premise of ensuring the output precision of the network, thereby improving the inference efficiency and energy efficiency in a resource-limited scene such as an edge device.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for skipping deep networks based on similarity detection. Background Technology

[0002] Symmetric encoder-decoder networks, exemplified by U-Net, have been widely applied in various fields such as image segmentation, object detection, semantic understanding, and signal reconstruction due to their multi-scale feature fusion capabilities. With the development of edge intelligence technology, U-Net and its variants are being deployed extensively in scenarios relying on continuous-time input, such as real-time video segmentation, dynamic medical image analysis, temporal sensor data parsing, and autonomous driving environmental perception. In these scenarios, input data is continuously generated in a temporal sequence, and adjacent input data often exhibit strong correlations, especially under conditions of stable motion, fixed background, or slow signal change, where the feature differences between adjacent inputs are minimal, demonstrating extremely high similarity.

[0003] However, the inference process of traditional encoder-decoder networks is fixed. Regardless of the similarity of the input data, the entire computational process, including multi-scale downsampling by the encoder, upsampling by the decoder, and cross-layer connection feature fusion, must be performed. Deep networks account for a very high proportion of computation, and repeatedly processing similar inputs causes a large amount of unnecessary computational overhead. Furthermore, frequent memory accesses further increase energy consumption and inference latency, severely restricting the deployment and application of U-Net networks in resource-constrained scenarios such as edge devices and real-time systems. Therefore, designing a dynamic computation optimization mechanism based on input similarity for encoder-decoder networks with continuous-time inputs has become a key requirement for improving their practical application performance.

[0004] Current optimization techniques also have certain limitations in application, as follows: 1. Static network pruning technique: The pruning strategy is static and fixed, which cannot adapt to the dynamic similarity changes of continuous time input. There is still redundant calculation for similar inputs, and excessive pruning can easily lead to accuracy loss.

[0005] 2. Fixed shallow inference scheme: This method directly presets the use of only shallow network inference in exchange for efficiency, lacks a dynamic judgment mechanism, and will cause a serious drop in output accuracy when the input difference is large.

[0006] 3. Simplified calculation method based on direct reuse: This method directly reuses the output of the previous frame for adjacent inputs, which not only easily leads to error accumulation, but also fails to optimize in conjunction with the network layer structure, resulting in a serious lack of flexibility and adaptability. Summary of the Invention

[0007] This application provides a method and apparatus for skipping deep network computations based on similarity detection. By evaluating the similarity of feature data input in consecutive time, it dynamically determines whether to skip deep network computations and only executes shallow network computation processes within a certain range, thereby optimizing computational resource consumption and processing efficiency while ensuring output accuracy.

[0008] This application provides a deep network skipping method based on similarity detection, applied to encoder-decoder networks. The deep network skipping method based on similarity detection includes: acquiring the current frame input data in a continuous time input sequence, and performing feature extraction using a shallow network in a predefined encoder-decoder network to obtain a current frame feature map; acquiring the most recently saved frame feature map in the feature cache unit, and calculating the average similarity between the current frame feature map and the most recently saved frame feature map; comparing the average similarity with a preset similarity threshold to obtain a similarity judgment result; and dynamically deciding whether to trigger a deep skipping instruction based on the similarity judgment result.

[0009] According to the deep network skipping method based on similarity detection provided in this application, the method dynamically decides whether to trigger a deep skip instruction based on the similarity judgment result, including: if the average similarity is greater than the similarity threshold, the deep skip instruction is triggered to skip the deep network calculation in the encoder-decoder network except for the shallow network range, and the output result is generated using the current frame feature map and the decoder corresponding to the shallow network.

[0010] According to the deep network skipping method based on similarity detection provided in this application, the method dynamically decides whether to trigger the deep skip instruction based on the similarity judgment result, including: if the average similarity is less than or equal to the similarity threshold, the deep skip instruction is not triggered, the encoder-decoder network is controlled to perform the whole process inference including deep network computation, the output result is generated, and the current frame feature map is stored in the feature cache unit.

[0011] According to the deep network skipping method based on similarity detection provided in this application, a skipping frequency constraint mechanism is further included. The constraint mechanism includes: setting a minimum skipping interval and / or a maximum skipping interval; wherein, the minimum skipping interval is used to define the minimum number of frames for continuously triggering deep skipping instructions; the maximum skipping interval is used to define the maximum number of frames for continuously triggering deep skipping instructions; when it is less than the minimum skipping interval, deep network computations in the encoder-decoder network, excluding the shallow network range, are forcibly skipped; when the maximum skipping interval is reached, the entire inference process, including deep network computations, is forcibly executed, and the feature cache unit is reset.

[0012] According to the deep network skipping method based on similarity detection provided in this application, before obtaining the current frame input data in the continuous time input sequence, the method further includes: defining the shallow network and deep network in the encoder-decoder network; the shallow network is a configurable network unit in the encoder-decoder network that has basic feature extraction and preliminary reconstruction capabilities; wherein, the layer range of the shallow network is configured as one or more fixed layers for single similarity detection; or configured as multiple shallow layers for multi-level similarity detection at different layers, and the layer range of the deep network to be skipped is determined according to the results of multi-level similarity detection.

[0013] According to the deep network skipping method based on similarity detection provided in this application, it further includes: determining a similarity threshold based on the layer range of the shallow network; reducing the similarity threshold when the defined shallow network is large; and increasing the similarity threshold when the defined shallow network range is small.

[0014] This application also provides a deep network skipping device based on similarity detection, applied to an encoder-decoder network. The deep network skipping device based on similarity detection includes: a shallow network extraction module, used to acquire the current frame input data in a continuous time input sequence, and to extract features using a shallow network in a predefined encoder-decoder network to obtain a current frame feature map; a similarity detection module, used to acquire the most recently saved frame feature map in a feature cache unit, and to calculate the average similarity between the current frame feature map and the most recently saved frame feature map; a threshold comparison module, used to compare the average similarity with a preset similarity threshold to obtain a similarity judgment result; and a dynamic execution module, used to dynamically decide whether to trigger a deep skip instruction based on the similarity judgment result.

[0015] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the deep network skipping method based on similarity detection as described above.

[0016] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep network skipping method based on similarity detection as described above.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the deep network skipping method based on similarity detection as described above.

[0018] This application provides a deep network skipping method and apparatus based on similarity detection, applied to encoder-decoder networks. The deep network skipping method based on similarity detection includes: acquiring the current frame input data from a continuous-time input sequence, and performing feature extraction using a pre-defined shallow network in the encoder-decoder network to obtain a current frame feature map; acquiring the most recently saved frame feature map from a feature cache unit, and calculating the average similarity between the current frame feature map and the most recently saved frame feature map; comparing the average similarity with a preset similarity threshold to obtain a similarity judgment result; and dynamically deciding whether to trigger a deep skipping instruction based on the similarity judgment result. Through this method, this application can dynamically skip redundant deep network calculations corresponding to similar inputs while ensuring network output accuracy, thereby improving inference efficiency and energy efficiency in resource-constrained scenarios such as edge devices. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the deep network skipping method based on similarity detection provided in the embodiments of this application.

[0021] Figure 2 This is a schematic diagram of deep skipping in the U-Net network provided in the embodiments of this application.

[0022] Figure 3 This is a schematic diagram of the overall process of the deep network skipping method based on similarity detection provided in the embodiments of this application.

[0023] Figure 4 This is a schematic diagram illustrating the deployment of the main calculations in the skipping method provided in this application embodiment.

[0024] Figure 5 This is a schematic diagram of the deep network skipping device based on similarity detection provided in the embodiments of this application.

[0025] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0028] The optimization techniques for the encoder-decoder network in the relevant solutions mainly include the following: 1. Static network pruning technique. This approach analyzes the network structure offline and prunes deep neurons, channels, or network branches that have little impact on output accuracy, thereby reducing the overall computational load. However, its core limitation lies in the fact that the pruning strategy is static and fixed, which cannot adapt to the dynamic similarity changes of continuous-time inputs. It cannot further reduce computation for similar inputs, and excessive pruning can easily lead to accuracy loss, resulting in poor versatility.

[0029] 2. Fixed Shallow Inference Scheme. This scheme is designed for specific scenarios and pre-sets to use only the shallow network of U-Net for inference, skipping all deep computations, sacrificing accuracy for efficiency. However, it lacks a dynamic judgment mechanism and cannot flexibly adjust the computation depth based on input similarity. When the inputs differ significantly, it leads to a severe drop in output accuracy and is difficult to adapt to complex and ever-changing continuous-time input scenarios.

[0030] 3. A simplified computation method based on direct reuse. This scheme directly reuses the output of the previous frame for adjacent inputs instead of performing network inference. However, its direct reuse of output results is prone to error accumulation and is not optimized for the hierarchical structure of the U-Net network, resulting in insufficient adaptability and flexibility.

[0031] The aforementioned solutions have significant limitations and cannot fully optimize the problem of redundant computation caused by similar inputs in continuously-time U-Net networks. Static network pruning techniques suffer from insufficient dynamic adaptability; the pruned network structure is fixed and cannot adjust the computation depth based on dynamic changes in input similarity. Redundant computation still exists for similar inputs, and its versatility is poor, requiring redesigned pruning strategies for specific tasks. It cannot adapt to various U-Net variants and different continuously-time input scenarios. Fixed shallow inference schemes suffer from the difficulty of balancing accuracy and efficiency, lacking a dynamic judgment mechanism and unable to flexibly switch computation depth based on input similarity. Significant accuracy loss occurs when input differences are large, failing to meet the accuracy requirements of complex scenarios. Simplified computation schemes based on direct reuse suffer from high error accumulation risk and poor adaptability. Directly reusing output results easily leads to error aggregation. Furthermore, they are not designed in conjunction with the hierarchical structure characteristics of U-Net networks, failing to selectively skip deep computations and only suitable for simple scenarios with extremely low accuracy requirements.

[0032] Based on this, embodiments of this application provide a deep network skipping method based on similarity detection, which can be applied to encoder-decoder networks, such as encoder-decoder networks that process continuous-time input sequences. Its specific objectives include at least one of the following: 1. Implement dynamic detection of the similarity of input data over continuous time to accurately determine whether deep network computation is required.

[0033] 2. Design a deep skipping mechanism that is compatible with various U-Net networks. The computation depth can be flexibly adjusted. When the input is similar, deep computation is skipped, and when the input is dissimilar, the entire computation process is performed, balancing accuracy and efficiency.

[0034] 3. Avoid error accumulation caused by directly reusing the output; ensure output accuracy by retaining shallow network calculations.

[0035] 4. It has strong versatility and can be adapted to various continuous-time input scenarios and U-Net variant networks without the need for customized architecture design for specific tasks.

[0036] Please see Figure 1 , Figure 1 This is a flowchart illustrating a deep network skipping method based on similarity detection provided in an embodiment of this application. In this embodiment, the deep network skipping method based on similarity detection may include steps S110 to S140, each step being as follows: S110: Obtain the current frame input data from the continuous time input sequence, and use the shallow network in the predefined encoder-decoder network to extract features and obtain the current frame feature map.

[0037] In the embodiments of this application, the encoder-decoder network refers to a deep neural network architecture widely used in image segmentation, video processing, and other fields. This architecture typically includes an encoder part and a decoder part, where the encoder is responsible for compressing the input data layer by layer into a high-dimensional feature representation, and the decoder is responsible for restoring the feature representation layer by layer into an output result with the same or close resolution as the input.

[0038] Optionally, the encoder-decoder network may include, for example, U-Net and its variants, but this application is not limited thereto. The encoder-decoder network may also be any other deep network with an encoder-decoder structure.

[0039] A continuous-time input sequence refers to an input data stream that is continuous in the time dimension.

[0040] For example, in video processing scenarios, a continuous time input sequence can be a series of frame images in a video stream; in autonomous driving scenarios, a continuous time input sequence can be continuous point cloud data or image data collected by sensors in chronological order.

[0041] Shallow networks and deep networks are two parts of an encoder-decoder network after being divided into layers. Optionally, a shallow network refers to a network layer in the encoder-decoder network that is close to the input and has the ability to extract basic features and perform preliminary reconstruction, such as extracting low-level features of an image, such as edges, textures, and colors.

[0042] Deep networks refer to network layers in encoder-decoder networks that are located after shallow networks and are responsible for extracting high-level semantic features. Their computational complexity is usually much higher than that of shallow networks.

[0043] The delineation of shallow and deep networks can be flexibly configured according to actual needs. For example, in a U-Net network with a 5-layer encoder and a 5-layer decoder, the first layer of the encoder and the last layer of the decoder can be designated as shallow networks, while the second to fifth layers of the encoder and the remaining layers of the decoder can be designated as deep networks.

[0044] When a new frame of data arrives in the continuous-time input sequence, this frame of data is used as the input data for the current frame and fed into the encoder-decoder network. The current frame input data does not go through all layers of the entire encoder-decoder network for processing; instead, it only undergoes feature extraction through a predefined shallow network.

[0045] Optionally, the shallow network performs basic feature extraction operations such as convolution and pooling on the current frame input data and outputs a feature map of the current frame. The current frame feature map can be, for example, one or more feature maps, which contain basic visual feature information such as edges, textures, and colors of the current frame input data.

[0046] S120: Obtain the most recently saved feature map in the feature cache unit and calculate the average similarity between the current frame feature map and the most recently saved frame feature map.

[0047] The feature cache unit is a cache module used to store feature maps of historical frames. It can be implemented as a hardware storage unit such as a register, memory buffer, or on-chip SRAM, or as a software-level data cache structure, which will not be listed here.

[0048] The most recently saved feature map in the feature cache unit refers to the feature map output by the shallow network and stored in the feature cache unit during the most recent full-process inference execution. When calculating the average similarity, the current frame feature map can be compared element-by-element with the most recently saved feature map. The average similarity can be calculated, for example, using cosine similarity. This involves calculating the cosine similarity between the feature vectors at corresponding positions in the current frame feature map and the most recently saved feature map, and then averaging the cosine similarities across all positions to obtain the average similarity.

[0049] Alternatively, the average similarity can also be calculated using the Structural Similarity Index (SSIM), the reciprocal of the Mean Squared Error (MSE), the Pearson correlation coefficient, etc., but this application is not limited to these methods. The average similarity can be calculated using any metric that can measure the overall similarity between two feature maps.

[0050] S130: Compare the average similarity with the preset similarity threshold to obtain the similarity judgment result.

[0051] The similarity threshold is a pre-set benchmark value used to determine whether the current frame is sufficiently similar to the most recently saved frame. The similarity threshold can be set to, for example, 0.95, but this application is not limited to this. The similarity threshold can be flexibly set according to the specific application scenario, accuracy requirements, and computing resource constraints.

[0052] The average similarity is compared numerically with the similarity threshold to obtain the similarity judgment result. The similarity judgment result can be, for example, "similar" (i.e., the average similarity is greater than the similarity threshold) or "dissimilar" (i.e., the average similarity is less than or equal to the similarity threshold).

[0053] S140: Based on the similarity judgment result, dynamically decide whether to trigger the deep skip instruction.

[0054] Based on the similarity assessment results, it is determined whether the current frame needs to undergo the full computation of the deep network. If the similarity assessment results indicate that the current frame is highly similar to the most recently saved frame, it means that the computation result of the deep network will not change significantly compared to the most recently saved frame. In this case, the computation of the deep network can be skipped to save computing resources and reduce inference latency.

[0055] If the similarity assessment results indicate that the current frame is not similar enough to the most recently saved frame, it means that the scene has changed significantly, and a complete deep network calculation needs to be performed to ensure output accuracy.

[0056] This dynamic decision-making mechanism based on inter-frame similarity can significantly reduce the computational overhead when processing continuous-time input sequences while ensuring output quality, making it particularly suitable for applications with high real-time requirements.

[0057] According to the deep network skipping method based on similarity detection provided in the embodiments of this application, the current frame feature map is extracted using only shallow networks and compared with the most recently saved frame feature map in the cache. Based on the comparison result, a dynamic decision is made on whether to skip deep network calculations. This method can effectively reduce unnecessary deep network calculations when the inter-frame changes of the input sequence in continuous time are small, thereby significantly reducing computational latency and energy consumption while ensuring output quality.

[0058] In some embodiments, the step of dynamically deciding whether to trigger a deep skip instruction based on the similarity judgment result may specifically include: If the average similarity is greater than the similarity threshold, a deep skip instruction is triggered, skipping the deep network calculations in the encoder-decoder network except for the shallow network range, and generating the output result using the current frame feature map and the decoder corresponding to the shallow network.

[0059] Specifically, when the average similarity is greater than the similarity threshold, it indicates that the shallow features of the current frame are highly similar to the shallow features of the most recently saved frame, meaning that the scene content presented in the current frame has not changed significantly compared to the most recently saved frame. In this case, a deep skip instruction is triggered.

[0060] The execution of the deep skip instruction means that the input data of the current frame is no longer processed by the deep network (i.e., the remaining network layers other than the shallow network) in the encoder-decoder network. Instead, the current frame feature map obtained in step S110 is directly used to input the current frame feature map into the decoder corresponding to the shallow network. The decoder then generates the output result of the current frame based on the current frame feature map.

[0061] For example, in a U-Net network with a 4-layer encoder (E1, E2, E3, E4) and a 4-layer decoder (D4, D3, D2, D1), if the first layer E1 of the encoder and the last layer D1 of the decoder are designated as shallow layers, and E2 to E4 and D4 to D2 are designated as deep layers, then when a deep skip instruction is triggered, the input data of the current frame only undergoes feature extraction through E1. The resulting feature map of the current frame is directly passed to D1 for decoding output, while the calculations of E2 to E4 and D4 to D2 are all skipped. In this way, the calculation that originally required 8 layers (E1→E2→E3→E4→D4→D3→D2→D1) is simplified to only 2 layers (E1→D1), significantly reducing the computational load.

[0062] This embodiment can significantly reduce the computational load and latency of processing each frame by skipping deep network computation when the inter-frame similarity is high. At the same time, since high inter-frame similarity means that the scene changes little, the difference between the output result generated by using only shallow features and the corresponding decoder and the output result of the whole process inference is small. Thus, it is possible to maintain an acceptable level of output quality while reducing computational overhead.

[0063] In some embodiments, the step of dynamically deciding whether to trigger a deep skip instruction based on the similarity judgment result may specifically include: If the average similarity is less than or equal to the similarity threshold, the deep skip instruction is not triggered. Instead, the encoder-decoder network is controlled to perform the entire inference process, including deep network computation, to generate the output result and store the current frame feature map in the feature cache unit.

[0064] In this embodiment, when the average similarity is less than or equal to the similarity threshold, it indicates that the current frame has undergone significant changes compared to the most recently saved frame, and the differences in shallow features have exceeded the range where deep computation can be safely skipped. In this case, the deep skip instruction is not triggered, meaning the encoder-decoder network is controlled to perform inference on the current frame input data according to the normal complete process.

[0065] Specifically, the input data of the current frame passes through all levels of the encoder (including the encoder parts in the shallow network and the deep network) and all levels of the decoder (including the decoder parts in the deep network and the corresponding decoder parts in the shallow network) in sequence, and finally generates a complete output result.

[0066] Simultaneously, after performing the full-process inference, the current frame feature map obtained in step S110 is stored in the feature cache unit to replace the most recently saved frame feature map in the feature cache unit. In this way, when processing the input data of the next frame, the feature cache unit stores the shallow feature map from the most recent full-process inference, which can be used as the similarity comparison benchmark for the next frame.

[0067] In this way, the feature cache unit always stores the shallow feature map corresponding to the most recent full-process inference, ensuring that the similarity comparison of subsequent frames has a reliable reference benchmark.

[0068] Compared with the technologies in related solutions, the embodiments of this application have the following significant advantages: 1) Achieving dynamic adaptation of computation depth, perfectly balancing accuracy and efficiency. This application's embodiments break through the limitations of static network pruning and fixed shallow inference. By introducing a similarity detection mechanism, it can accurately skip a large amount of redundant computation in deep networks when the input data is highly similar, and automatically resume the entire inference process when the input differences are large, achieving a dynamic balance between efficiency optimization and high-precision output.

[0069] 2) Effectively eliminates the problem of error accumulation and improves output stability. Compared with the coarse strategy of directly reusing the output results of the previous frame, the embodiments of this application retain the feature extraction and calculation process of the shallow network, and generate new outputs through real shallow features, thus avoiding the error superposition effect caused by continuous frame reuse.

[0070] 3) It possesses strong scene generalization ability and hardware compatibility. The shallow network boundary, similarity threshold, and skip spacing implemented in this application all support dynamic parameterization configuration through software and hardware collaboration. At the same time, the similarity detection and threshold comparison logic can be directly and seamlessly compatible with the existing computing units of conventional AI accelerators (such as GPUs, FPGAs, etc.) without the need for additional costs to customize the underlying hardware.

[0071] In some embodiments, the deep network skipping method based on similarity detection may further include a configuration skipping frequency constraint mechanism, which limits the triggering frequency of deep skipping instructions to avoid problems such as skipping too many consecutive frames or not skipping too many consecutive frames in extreme cases, thereby achieving a better balance between computational efficiency and output quality.

[0072] Constraint mechanisms may include setting a minimum skip gap and / or a maximum skip gap.

[0073] Specifically, when the skip interval is less than the minimum skip interval, deep network computations in the encoder-decoder network, excluding those in the shallow network range, are forcibly skipped; when the maximum skip interval is reached, the entire inference process, including deep network computations, is forcibly executed, and the feature cache unit is reset.

[0074] In this embodiment, the minimum skip interval is used to define the minimum number of frames required to trigger a deep skip instruction consecutively. For example, if the minimum skip interval is set to 2, it means that the deep skip instruction must be triggered at least twice consecutively.

[0075] When the distance is less than the minimum skip gap, deep network computations in the encoder-decoder network, excluding shallow network layers, are forcibly skipped. That is, regardless of whether the average similarity between the current frame and the most recently saved frame is greater than a similarity threshold, a skip operation is forced as long as the frame interval since the last full-process inference has not reached the minimum skip gap. This mechanism effectively avoids unnecessary computational overhead caused by frequently performing full-process inference in scenarios with extremely small inter-frame changes.

[0076] The maximum skip interval is used to define the maximum number of frames that can be consecutively triggered by a deep skip instruction. For example, if the maximum skip interval is set to 5, it means that the number of consecutive skipped frames cannot exceed 5 frames, that is, after every 5 consecutive skipped frames, at least one full-process inference must be performed.

[0077] When the maximum skip interval is reached, a full inference process, including deep network computation, is forcibly executed, and the feature cache unit is reset. That is, regardless of whether the average similarity between the current frame and the most recently saved frame is greater than the similarity threshold, as long as the number of consecutive skipped frames has reached the maximum skip interval, a full inference process is forcibly executed to prevent the gradual accumulation of deviations between the output results and the actual scene due to prolonged periods without complete computation. Simultaneously, after forcibly executing the full inference process, the feature cache unit is reset; that is, the shallow feature map of the current frame is stored in the feature cache unit as a similarity comparison benchmark for subsequent frames. By resetting the feature cache unit, it is ensured that the similarity comparison of subsequent frames is not affected by too early historical frames.

[0078] It should be understood that the above values ​​are only examples, and the specific values ​​of the minimum and maximum skip intervals can be flexibly configured according to the real-time and accuracy requirements of the actual application scenario.

[0079] It should be noted that the minimum skip interval and the maximum skip interval can be set simultaneously, or only one of them can be set; this application is not limited to this.

[0080] In some embodiments, the steps prior to obtaining the current frame input data in the continuous-time input sequence may further include: Define the shallow and deep networks in the encoder-decoder network.

[0081] The shallow network is a configurable network unit in the encoder-decoder network that has the ability to extract basic features and perform preliminary reconstruction.

[0082] The number of layers in a shallow network can be fixed or flexibly configured based on factors such as network structure, application scenario, accuracy requirements, and computing resources.

[0083] According to one example of an embodiment of this application, the layer range of a shallow network is configured to be a fixed number of one or more layers for single similarity detection.

[0084] In this configuration, feature maps are extracted and similarity comparisons are performed only at a predetermined shallow layer location. The result of this single comparison determines whether to skip all deep network computations from that layer to the network output layer.

[0085] For example, in a network with a 5-layer encoder and a 5-layer decoder, the first layer of the encoder can be designated as a shallow layer. At this layer, the feature map of the current frame is extracted and compared with the feature map of the most recently saved frame in the cache. If the similarity is greater than a threshold, the computation of layers 2 to 5 of the encoder and layers 5 to 2 of the decoder is skipped, and the output result is generated directly using the feature map of the current frame and layer 1 of the decoder.

[0086] For example, the first and second layers of the encoder can be defined as a shallow network. A single similarity detection is performed at the output position of the second layer. If the skip condition is met, the calculation of the third to fifth layers of the encoder and the corresponding decoder layers is skipped.

[0087] According to another example of the embodiments of this application, the layer range of the shallow network can also be configured as multiple shallow layers to perform multi-level similarity detection at different layers, and the layer range of the deep network to be skipped is determined based on the results of the multi-level similarity detection.

[0088] In this configuration, similarity detection is performed at multiple shallow layers of the encoder-decoder network. The similarity detection results at each layer can be used to determine whether to skip the deep network computation after that layer.

[0089] For example, in a network with a 5-layer encoder and a 5-layer decoder, similarity detection points can be set in layers 1 and 2 of the encoder. First, a first-level similarity detection is performed in layer 1. If the average similarity of layer 1 is greater than the corresponding first-level similarity threshold, all calculations for layers 2 through 5 of the encoder and their corresponding decoder layers are skipped. If the average similarity of layer 1 is less than or equal to the first-level similarity threshold, the calculations for layer 2 are continued, and a second-level similarity detection is performed in layer 2. If the average similarity of layer 2 is greater than the corresponding second-level similarity threshold, the calculations for layers 3 through 5 of the encoder and their corresponding decoder layers are skipped. If the average similarity of layer 2 is also less than or equal to the second-level similarity threshold, the entire inference process is executed.

[0090] By using multi-level similarity detection, finer-grained deep skip control can be achieved. This means that different ranges of deep network layers can be skipped adaptively based on the degree of inter-frame variation, thereby achieving a better balance between computational efficiency and output accuracy.

[0091] In some embodiments, the steps of the deep network skipping method based on similarity detection may further include: The similarity threshold is determined based on the layer range of the shallow network; when the defined shallow network is large, the similarity threshold is reduced; when the defined shallow network range is small, the similarity threshold is increased.

[0092] In this embodiment, the similarity threshold is related to the number of layers in the shallow network. When the defined shallow network is large (i.e., the shallow network contains more layers and extracts richer features), the similarity threshold is reduced. This is because when the shallow network is large, its extracted feature maps already contain sufficient scene information. Even if the average similarity is slightly low, the decoder corresponding to the shallow network can still generate high-quality output results based on these sufficient features. Therefore, the skip condition can be appropriately relaxed, i.e., the similarity threshold is reduced, to obtain a higher skip rate and lower computational cost.

[0093] When the defined shallow network scope is small (i.e., the shallow network contains fewer layers and extracts simpler features), the similarity threshold should be increased. This is because when the shallow network is small, its extracted feature maps only contain very basic low-level features. When the decoder corresponding to the shallow network decodes based on these limited features, the quality of the output is more sensitive to inter-frame differences. Therefore, it is necessary to increase the similarity threshold so that skipping is only triggered when the frames are extremely similar, in order to ensure output quality.

[0094] For example, when the shallow network contains the first 3 layers of the encoder, the similarity threshold can be set to 0.90; when the shallow network contains only the first 1 layer of the encoder, the similarity threshold can be set to 0.98.

[0095] It should be understood that the above values ​​are merely examples, and the specific value of the similarity threshold can be adjusted according to the actual network structure, application scenario, and accuracy requirements.

[0096] In this embodiment, by adaptively determining the similarity threshold based on the layer range of the shallow network, the deep skipping mechanism can achieve a good balance between computational efficiency and output quality under different shallow network configurations.

[0097] The deep network skipping method based on similarity detection in the above embodiments divides the encoder-decoder network into shallow and deep networks, uses feature maps extracted by the shallow network for inter-frame similarity detection, and dynamically decides whether to skip deep network computation based on the similarity judgment result. This effectively reduces redundant deep computation when processing continuous-time input sequences with small inter-frame variations, significantly improving inference efficiency and reducing computational latency and energy consumption while ensuring output quality. Furthermore, by configuring a skipping frequency constraint mechanism and adaptively adjusting the similarity threshold according to the shallow network layer range, the robustness and flexibility of this method in different application scenarios can be further enhanced.

[0098] In summary, this application addresses the problem of repetitive computation in U-Net and its variants when processing similar input data in continuous-time input scenarios. In various tasks relying on continuous-time input (such as video frame sequences, time-series sensor data, and continuous signal sampling), U-Net networks typically need to perform complete multi-layer network computations for each frame of input. However, in practical applications, when the motion state of the input data is stable, the background does not change significantly, or the interval between adjacent inputs meets certain conditions, adjacent input data often exhibits extremely high similarity. Related U-Net networks lack dynamic computation optimization mechanisms for such similar inputs, and still repeatedly perform deep computations throughout the entire process, leading to wasted computational resources, increased inference latency, and increased energy consumption. This problem is particularly pronounced in resource-constrained scenarios such as edge devices.

[0099] This application aims to provide a general solution that, while ensuring the accuracy of network output, dynamically skips deep network computations corresponding to similar inputs, thereby improving the inference efficiency and energy efficiency of the U-Net network in continuous-time input scenarios.

[0100] Please see Figure 2 , Figure 2 This is a schematic diagram of deep skipping in the U-Net network provided in the embodiments of this application.

[0101] This application embodiment optimizes computational load and efficiency while ensuring output accuracy by evaluating the similarity of continuously input feature data and dynamically deciding whether to skip the deep computation of the U-Net network, retaining only a certain range of shallow network computation processes.

[0102] Among them, the selection of shallow networks can be flexibly defined according to the actual situation such as U-Net network structure, task accuracy requirements, and hardware resource constraints, and the range of suitable shallow layers can be determined, which has strong scene adaptability.

[0103] U-Net is mainly divided into two parts: an encoder and a decoder (i.e., a decoder). The encoder extracts features through downsampling, while the decoder enlarges the feature map size through residual modules, upsampling, etc., and finally generates an output result with the same size as the input.

[0104] In this embodiment, a similarity detection module is inserted between the encoder's feature extraction and the decoder's result generation to determine whether to skip the deep network.

[0105] The encoder's feature extraction is used to extract key feature maps from continuous-time input data, providing a basis for similarity evaluation. The feature extraction level is the shallow level, which is always on during computation. Deeper levels are triggered by similarity.

[0106] like Figure 2 As shown, Level 0 is the shallow level, and Level 1, Level 2, and Level 3 are the deep levels.

[0107] The similarity detection module calculates the average similarity between adjacent input feature maps and outputs a similarity judgment result. When the similarity is higher than a set threshold, only the shallow network is activated to perform calculations, skipping deeper layers such as encoder downsampling, decoder upsampling, and corresponding cross-layer connection calculations. When the similarity is lower than the threshold, the deep network is triggered to perform the entire calculation process to ensure output accuracy.

[0108] In this embodiment, the shallow network is defined as a configurable network unit in the U-Net network that possesses basic feature extraction and preliminary reconstruction capabilities. Its layer range can be flexibly adjusted—it can be a fixed encoder downsampling and corresponding decoder upsampling structure for one or several layers, performing a single similarity detection after the encoder computation at these layers is complete; or it can be expanded into multiple shallow layers according to complex task requirements, enabling multi-level similarity detection at different layers, and selecting the skipped deep network range based on the similarity detection results. The core logic for selecting the shallow network is to adapt to the balance between accuracy and computational efficiency in different scenarios.

[0109] Similarity detection can employ an average similarity (AS) calculation method based on pixel-level differences, ultimately yielding the skip enable logic. The specific formula is as follows: ; ; in, and They respectively represent the first Frame and the The frame input contains the numerical values ​​of the feature map at position (x,y). This represents the number of elements in the entire feature map. The threshold value used for comparison with the average similarity.

[0110] It should be noted that the calculation method in this application is not limited to the average similarity calculation described above; it is merely illustrative. Optionally, the similarity detection calculation method can be adapted to various continuous-time input feature maps and corresponding U-Net and its variant networks. The threshold can be dynamically adjusted according to the accuracy requirements of different tasks, and combined with the definition of the shallow layer range, a flexible balance between accuracy and efficiency can be achieved.

[0111] The method described in this application is applicable to U-Net and its variants for processing continuous-time input sequences. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the overall process of the deep network skipping method based on similarity detection provided in the embodiments of this application. The overall process includes steps such as off-chip access and initialization configuration, network deployment and calculation, and deep skipping judgment.

[0112] First, perform external access and initialization configuration. This mainly involves the following steps: 1. Define the range of suitable shallow networks. Based on the U-Net network structure, task accuracy requirements, and hardware resource constraints, define the encoder downsampling layers and corresponding decoder upsampling layers included in the shallow network to ensure that the shallow network can meet the basic feature extraction and output generation capabilities while also adapting to the scene efficiency requirements.

[0113] 2. Set a similarity threshold. It can be dynamically adjusted according to the accuracy requirements of the task, and the threshold can be adjusted in conjunction with the shallow range.

[0114] 3. Initialize the on-chip cache unit and transfer the set parameters, feature map data, weight data, etc. to the on-chip cache through off-chip access (EMA).

[0115] Secondly, network deployment and computing are carried out. This mainly involves the following steps: For the i-th frame of input data in a continuous-time input sequence, the feature map can be extracted using the encoder weights of each layer in the U-Net network. It possesses the core representational information of the input data.

[0116] Correspondingly, the output can also be generated through the weights of each decoder layer in the U-Net network. If the input is the first frame of the sequence (i=0), there is no need to perform similarity detection; the entire network inference process is executed directly to generate the output and the feature map is then processed. Stored in the feature buffer unit, and proceed to the next frame processing flow.

[0117] Next, a deep skip check is performed. This mainly involves the following steps: The process begins by determining the current level (Level determination). If the set shallow boundary has not yet been reached, the parameters for the next level are retrieved and calculation continues. Otherwise, similarity detection is performed, and the saved feature map is read from the feature cache unit for calculation. and The average similarity AS.

[0118] The calculated average similarity AS is compared with the threshold. For comparison, if AS reaches If the current input is determined to be highly similar to the saved frame, a deep skip instruction is triggered. The corresponding decoder parameters of the current layer are obtained and calculated, skipping deep downsampling of the encoder, deep upsampling of the decoder, and corresponding cross-layer connection calculations outside the shallow layer range. Otherwise, if the current input is determined to be significantly different from the most recently saved frame, the skip instruction is not triggered, and the network is controlled to execute the entire inference process to ensure the integrity of multi-scale feature fusion.

[0119] The skipping method in this application embodiment can be deployed in hardware to implement the complete process. To ensure data flow compatibility, the main computation of this skipping method needs to be adapted to the computing architecture and storage logic of conventional AI accelerators (such as GPUs, FPGAs, ASICs, and general-purpose deep learning accelerators). Through modular design and standardized configuration, the deep skipping judgment calculation, weight management, feature map processing, and threshold comparison operations are seamlessly compatible with the conventional accelerator process.

[0120] This application's embodiments require no additional customized hardware and can be implemented solely through software configuration. Compatibility deployment mainly includes similarity detection and threshold comparison calculations.

[0121] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the deployment of the main calculations in the skipping method provided in this application embodiment.

[0122] Similarity detection can be performed by comparing the cached feature map from the feature cache module with the current frame's feature map. Pixel-level difference calculations can be completed in the accelerator by configuring the weights to 1 / -1. Threshold comparison operations also require no additional hardware; the comparator or ALU in the accelerator can be reused directly. The judgment result is output as a binary signal (1 for high similarity, 0 for low similarity).

[0123] It should be noted that this application is not limited to the pixel-level difference calculation and threshold comparison methods described in the figures and text; these are merely illustrative examples.

[0124] Furthermore, to achieve precise and controllable deep skipping mechanism, this application's embodiments construct a hardware-software collaborative parameterized design system around three core parameters: shallow network selection, similarity threshold setting, and minimum / maximum skipping interval. Through deep linkage between precise software parameter definition and dynamic hardware resource adaptation, it ensures flexibility in parameter adjustment while avoiding problems such as frequent switching and error accumulation. Specifically, this includes the following: 1. Shallow Network Selection and Design: The shallow network selection parameters define the core computational range that can skip deep computations, forming the basis for balancing accuracy and efficiency. Multiple layer configurations are employed to achieve hardware and software co-adaptation. Shallow layers can be fixed (one or several layers), with single similarity detection performed after the encoder computation at these layers; or they can be divided into multiple shallow layers, performing multi-level similarity detection at different levels. Different shallow network selections will result in different computational loads, EMA values, and accuracy losses, allowing for fine-grained design based on actual task requirements.

[0125] 2. Similarity threshold setting: Similarity threshold This threshold is used to determine whether to trigger a deep skip, and it is optimized in conjunction with the shallow layer selection parameters to adapt to different accuracy requirements and accelerator computational precision. When the shallow layer range is selected to be large, the threshold can be appropriately reduced to increase the skipping frequency and reduce the overall computational cost; when the shallow layer range is selected to be small, the threshold can be appropriately increased to avoid excessive skipping that could lead to accuracy loss.

[0126] 3. Minimum / Maximum Skip Spacing Parameter Design: Design the minimum skip spacing (D) min ) and maximum jump interval (D max ) parameters, construct a skip frequency constraint mechanism, and achieve collaborative management and control of software and hardware. D min Define the minimum number of frames required for consecutive deep skip instructions to avoid accelerator scheduling overhead caused by frequent switching of computing modes within a short period of time; D max Defines the maximum number of frames for consecutively triggering deep skip instructions, forces a full-process calculation to be performed at certain frame intervals, and resets the feature cache and error accumulation. Parameters can be customized according to the scenario and can also be linked with shallow range.

[0127] Through the above embodiments, the embodiments of this application have dynamic adaptation capabilities and flexible configuration. They do not require offline pruning, can dynamically adjust the calculation depth according to the similarity of continuous input data, and can flexibly define the shallow layer range according to the needs of the scenario. When similar inputs are received, deep calculations are skipped, further reducing redundant calculations. Moreover, the core network structure is not changed, and optimization is achieved only by dynamically controlling the calculation range.

[0128] This application's embodiments achieve both dynamic switching of computation depth and break the fixed limitation of shallow layer range. On the one hand, through similarity detection, full-process computation is performed to ensure accuracy when input differences are large, and deep computation is skipped to improve efficiency when inputs are highly similar, thus balancing the contradiction between accuracy and efficiency. On the other hand, the shallow layer range can be adjusted according to task and hardware requirements to adapt to complex and ever-changing continuous-time input scenarios and resource constraints, solving the dual problems of severe accuracy loss and poor adaptability of fixed shallow layer schemes.

[0129] This application generates new outputs by retaining a configurable range of shallow network computations instead of directly reusing the results of the previous frame, thus avoiding error accumulation and significantly improving the stability of output accuracy. At the same time, combined with the U-Net network hierarchical structure design, it enables selective skipping of deep computations instead of completely skipping network inference, and the shallow range can be flexibly adjusted, making it more applicable and more flexible.

[0130] Furthermore, this embodiment has been trained and simulated on a specific dataset, and deployed and tested on a standard 28nm chip. The performance metrics after deployment on the chip are shown below:

[0131] Among them, after adopting the solution of the embodiments of this application, the on-chip energy efficiency is 35.80 TOPS / W under a specific dataset, the system energy efficiency considering EMA cost is 14.07 TOPS / W, the average energy overhead per inference is 2.07-2.88 mJ, and the average latency overhead per inference is 58.53-79.95 ms. Compared with the baseline results without using the solution of the embodiments of this application, the EMA is reduced by 76%, the energy overhead is reduced by 77%, and the latency is reduced by 77% under a specific dataset.

[0132] This application also provides a deep network skipping device based on similarity detection. The deep network skipping device based on similarity detection provided in this application will be described below. The deep network skipping device based on similarity detection described below can be referred to in correspondence with the deep network skipping method based on similarity detection described above.

[0133] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a deep network skipping device based on similarity detection provided in an embodiment of this application. In this embodiment, the deep network skipping device based on similarity detection can be applied to an encoder-decoder network. The deep network skipping device based on similarity detection may include a shallow network extraction module 510, a similarity detection module 520, a threshold comparison module 530, and a dynamic execution module 540.

[0134] The shallow network extraction module 510 is used to acquire the current frame input data in the continuous time input sequence and perform feature extraction using the shallow network in the predefined encoder-decoder network to obtain the current frame feature map.

[0135] The similarity detection module 520 is used to obtain the most recently saved frame feature map in the feature cache unit and calculate the average similarity between the current frame feature map and the most recently saved frame feature map.

[0136] The threshold comparison module 530 is used to compare the average similarity with a preset similarity threshold to obtain a similarity judgment result.

[0137] The dynamic execution module 540 is used to dynamically decide whether to trigger the deep skip instruction based on the similarity judgment result.

[0138] In some embodiments, the dynamic execution module 540 is specifically used for: If the average similarity is greater than the similarity threshold, a deep skip instruction is triggered, skipping the deep network calculations in the encoder-decoder network except for the shallow network range, and generating the output result using the current frame feature map and the decoder corresponding to the shallow network.

[0139] In some embodiments, the dynamic execution module 540 is specifically used for: If the average similarity is less than or equal to the similarity threshold, the deep skip instruction is not triggered. Instead, the encoder-decoder network is controlled to perform the entire inference process, including deep network computation, to generate the output result and store the current frame feature map in the feature cache unit.

[0140] In some embodiments, the deep network skipping device based on similarity detection further includes a constraint mechanism configuration module, which is used to: Configure a skip frequency constraint mechanism, which includes: setting a minimum skip interval and / or a maximum skip interval; wherein, the minimum skip interval is used to define the minimum number of frames for consecutively triggering deep skip instructions; the maximum skip interval is used to define the maximum number of frames for consecutively triggering deep skip instructions; when it is less than the minimum skip interval, deep network computations in the encoder-decoder network, excluding the shallow network range, are forcibly skipped; when the maximum skip interval is reached, the entire inference process, including deep network computations, is forcibly executed, and the feature cache unit is reset.

[0141] In some embodiments, the deep network skipping device based on similarity detection further includes a network partitioning module, which is used to: Define the shallow and deep networks in the encoder-decoder network; the shallow network is a configurable network unit in the encoder-decoder network that has basic feature extraction and preliminary reconstruction capabilities; the shallow network can be configured with one or more fixed layers for single similarity detection; or it can be configured with multiple shallow layers for multi-level similarity detection at different layers, and the layer range of the deep network to be skipped is determined based on the results of multi-level similarity detection.

[0142] In some embodiments, the deep network skipping device based on similarity detection further includes a threshold determination module, which is used to: The similarity threshold is determined based on the layer range of the shallow network; when the defined shallow network is large, the similarity threshold is reduced; when the defined shallow network range is small, the similarity threshold is increased.

[0143] On the other hand, this application also provides an electronic device, please refer to... Figure 6 , Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device may include memory 620, processor 610, and a computer program stored in memory 620 and executable on processor 610. When processor 610 executes the program, it can implement a deep network skipping method based on similarity detection, which may include: The system acquires the current frame input data from the continuous-time input sequence and performs feature extraction using a shallow network in a predefined encoder-decoder network to obtain the current frame feature map. It then acquires the most recently saved frame feature map from the feature cache unit and calculates the average similarity between the current frame feature map and the most recently saved frame feature map. The average similarity is compared with a preset similarity threshold to obtain the similarity judgment result. Based on the similarity judgment result, the system dynamically decides whether to trigger the deep skip instruction.

[0144] Optionally, the electronic device may further include a communication bus 630 and a communication interface 640, wherein the processor 610, the communication interface 640, and the memory 620 communicate with each other through the communication bus 630. The processor 610 may call the computer program in the memory 620 to execute the deep network skipping method based on similarity detection provided by the above methods.

[0145] Furthermore, the logical instructions in the aforementioned memory 620 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the deep network skipping method based on similarity detection provided by the above methods. The steps and principles of the method have been described in detail in the above methods and will not be repeated here.

[0147] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it is implemented to perform the deep network skipping method based on similarity detection provided by the above methods. The steps and principles of the method have been described in detail in the above methods and will not be repeated here.

[0148] Non-transitory computer-readable storage media can be any available medium or data storage device that can be accessed by a processor, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A deep network skipping method based on similarity detection, characterized in that, Applied to encoder-decoder networks, the deep network skipping method based on similarity detection includes: The current frame input data in the continuous time input sequence is obtained, and the shallow network in the predefined encoder-decoder network is used to extract features to obtain the current frame feature map. Obtain the most recently saved feature map in the feature cache unit, and calculate the average similarity between the current frame feature map and the most recently saved frame feature map; The average similarity is compared with a preset similarity threshold to obtain a similarity judgment result; Based on the similarity judgment result, a dynamic decision is made as to whether to trigger the deep skip instruction.

2. The deep network skipping method based on similarity detection according to claim 1, characterized in that, The step of dynamically deciding whether to trigger a deep skip instruction based on the similarity judgment result includes: If the average similarity is greater than the similarity threshold, the deep skip instruction is triggered to skip the deep network calculations in the encoder-decoder network except for the shallow network range, and the output result is generated using the current frame feature map and the decoder corresponding to the shallow network.

3. The deep network skipping method based on similarity detection according to claim 1, characterized in that, The step of dynamically deciding whether to trigger a deep skip instruction based on the similarity judgment result includes: If the average similarity is less than or equal to the similarity threshold, the deep skip instruction is not triggered, the encoder-decoder network is controlled to perform full-process inference including deep network computation, generate output results, and store the current frame feature map into the feature cache unit.

4. The deep network skipping method based on similarity detection according to claim 1, characterized in that, It also includes configuring a skip frequency constraint mechanism, the constraint mechanism including: Set a minimum skip interval and / or a maximum skip interval; wherein, the minimum skip interval is used to define the minimum number of frames for consecutively triggering deep skip commands; the maximum skip interval is used to define the maximum number of frames for consecutively triggering deep skip commands; When the distance is less than the minimum skip interval, deep network calculations in the encoder-decoder network, excluding the shallow network range, are forcibly skipped. When the maximum skip interval is reached, a full-process inference including deep network computation is forcibly executed, and the feature cache unit is reset.

5. The deep network skipping method based on similarity detection according to any one of claims 1 to 4, characterized in that, Before acquiring the current frame input data in the continuous-time input sequence, the method further includes: Define the shallow network and deep network in the encoder-decoder network; the shallow network is a configurable network unit in the encoder-decoder network that has basic feature extraction and preliminary reconstruction capabilities; The shallow network can be configured with one or more fixed layers for single similarity detection, or with multiple shallow layers for multi-level similarity detection at different layers, and the range of layers of the deep network to be skipped can be determined based on the results of the multi-level similarity detection.

6. The deep network skipping method based on similarity detection according to claim 5, characterized in that, Also includes: The similarity threshold is determined based on the hierarchical range of the shallow network; When the defined shallow network is large, the similarity threshold is reduced; When the defined shallow network range is small, the similarity threshold is increased.

7. A deep network skipping device based on similarity detection, characterized in that, The deep network skipping device based on similarity detection, applied to encoder-decoder networks, includes: The shallow network extraction module is used to acquire the current frame input data in the continuous time input sequence and perform feature extraction using the shallow network in the predefined encoder-decoder network to obtain the current frame feature map. The similarity detection module is used to obtain the most recently saved frame feature map in the feature cache unit and calculate the average similarity between the current frame feature map and the most recently saved frame feature map; The threshold comparison module is used to compare the average similarity with a preset similarity threshold to obtain a similarity judgment result; The dynamic execution module is used to dynamically decide whether to trigger the deep skip instruction based on the similarity judgment result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the deep network skipping method based on similarity detection as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the deep network skipping method based on similarity detection as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the deep network skipping method based on similarity detection as described in any one of claims 1 to 6.

Images