Low-resolution object target detection method and system based on heterogeneous calculation

Abstract

The invention relates to the technical field of image recognition, in particular to a low-resolution object target detection method and system based on heterogeneous computation.In the technical scheme, firstly, the energy consumption level of a current system is obtained, and a recognition strategy of the corresponding level is called according to the energy consumption level of the current system; therefore, the called identification strategy does not occupy large energy consumption of the current system. And then, according to the resolution level of the to-be-processed image, the execution efficiency of the recognition strategy is adjusted, and the system energy consumption during target recognition execution is further reduced. Moreover, the execution efficiency of the recognition strategy is adjusted according to the resolution level of the to-be-processed image, and the processing efficiency of the to-be-processed image can be ensured, thereby achieving the balance of efficiency and energy consumption.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a low-resolution object detection method and system based on heterogeneous computing. Background Art

[0002] In the prior art, the following technical solutions are mainly used for low-resolution object detection: CPU solution: Currently, many low-resolution object detection systems use general-purpose CPUs for image processing and object detection. The advantage of CPUs is their versatility and programming flexibility, which can handle a variety of tasks. However, CPUs are less efficient when processing image data, especially when processing a large number of low-resolution images, with slow calculation speeds and high energy consumption. This makes CPU solutions perform poorly in scenarios that require long-term stable operation, such as industrial monitoring, deep space exploration, etc. For example, in deep space exploration missions, energy supply is limited, and low-energy solutions are needed to extend the service life of the equipment.

[0003] GPU solution: GPU performs well in parallel computing and can significantly improve image processing speed. However, GPU has high energy consumption and is not suitable for long-term operation scenarios, such as industrial monitoring and deep space exploration. For example, in the automatic identification task of unmanned ships, low-energy solutions are required to work continuously for hours or even days, otherwise the battery will need to be replenished frequently, affecting the continuity and reliability of the task.

[0004] FPGA solution: FPGA can implement specific computing tasks at the hardware level with relatively low energy consumption, but it has high development complexity, poor flexibility, and difficulty in quickly deploying new models and algorithms. This is an obvious disadvantage for scenarios that require rapid iteration and updating (such as laboratory research). In addition, FPGAs occupy a large amount of resources, which is not conducive to miniaturization and portability. Summary of the invention

[0005] In order to at least overcome to some extent the problem in the related art that it is difficult to achieve a balance between efficiency and energy consumption for target detection of low-resolution objects, the present application provides a low-resolution object detection method and system based on heterogeneous computing.

[0006] The scheme of this application is as follows: According to a first aspect of an embodiment of the present application, a low-resolution object detection method based on heterogeneous computing is provided, comprising: Receive image processing requests and images to be processed sent by users; Determine the resolution level of the image to be processed, and obtain the energy consumption level of the current system; Call the corresponding level identification strategy according to the energy consumption level of the current system; Based on the called recognition strategy, adjusting the execution efficiency of the recognition strategy according to the resolution level of the image to be processed as the execution strategy; Performing low-resolution object detection on the image to be processed by the execution strategy, and outputting an object detection image; The levels of the identification strategy include at least: a primary identification strategy, a secondary identification strategy, and a tertiary identification strategy; The recognition algorithms adopted by the recognition strategies of different levels are different. The energy consumption and processing capability of the first-level recognition strategy are greater than those of the second-level recognition strategy and greater than those of the third-level recognition strategy.

[0007] Preferably, the method further comprises: Determining whether the image processing request includes a recognition strategy level required by a user; If the image processing request includes a recognition strategy level required by the user, determining whether the recognition strategy level required by the user exceeds the currently called recognition strategy level; If the identification strategy level requested by the user does not exceed the currently called identification strategy level, the current execution strategy is maintained; If the recognition strategy level required by the user exceeds the recognition strategy level currently called, determine whether the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold; If the execution efficiency corresponding to the current execution strategy does not exceed the set efficiency threshold level, the execution efficiency of the identification strategy is adjusted to the maximum value, and the execution strategy is updated; If the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold, the execution strategy is updated according to the identification strategy level required by the user.

[0008] Preferably, the method further comprises: If the identification strategy level required by the user exceeds the preset identification strategy level, determining whether the current system energy consumption level is lower than the preset energy consumption threshold; If the energy consumption level of the current system is not lower than the preset energy consumption threshold, determine whether the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level identification strategy; If the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level recognition strategy, then based on the full-efficiency execution of the first-level recognition strategy, the second-level recognition strategy is synchronously executed with full efficiency to perform low-resolution object target detection on the image to be processed; If the current execution strategy is a first-level recognition strategy, but the corresponding execution efficiency is not the maximum value of the first-level recognition strategy, then on the basis of the full-efficiency execution of the first-level recognition strategy, the third-level recognition strategy is synchronously executed with full efficiency to perform low-resolution object target detection on the image to be processed.

[0009] Preferably, the method further comprises: If the energy consumption level of the current system is lower than the preset energy consumption threshold, all recognition strategies are synchronously executed with full efficiency to perform low-resolution object detection on the image to be processed.

[0010] Preferably, the method further comprises: Assign weights to each identification strategy; According to the weights assigned to each recognition strategy, the target detection results of all recognition strategies are fused and the consistency detection results are output.

[0011] Preferably, the method further comprises: Obtaining the parameters of the current display system, and determining the display image format according to the parameters of the current display system; Converting the target detection image into the display image format; The target detection image in the display image format is enhanced by histogram equalization and noise is filtered out by Gaussian filtering; Determine the resolution based on the parameters of the current display system; The resolution of the processed target detection image is adjusted according to the resolution of the current display system by bilinear interpolation; The target detection image with adjusted resolution is output through the current display system.

[0012] Preferably, the primary identification strategy includes: Based on the parallel computing capability and high-bandwidth memory access capability of the GPU, low-resolution object target detection is performed on the image to be processed by a first recognition algorithm; The first recognition algorithm comprises: Capturing the global dependencies of the image to be processed through the self-attention mechanism of the Transformer encoder; According to the global dependency of the image to be processed, noise is gradually added to the image to be processed. After T steps, a completely randomized feature map is obtained. Step by step, reduce the noise in the completely randomized feature map, and after T steps, restore the image to be processed; Extract features at different scales from the image to be processed after adding noise and reverse denoising, and fuse the extracted features; different weights are assigned to each scale; Perform low-resolution object detection on the image after feature fusion and output the target detection image.

[0013] Preferably, the secondary identification strategy includes: Based on ARM's multi-core parallel structure, the frequency can be dynamically adjusted according to different loads; Performing low-resolution object detection on the image to be processed by a second recognition algorithm; The second recognition algorithm comprises: Performing hierarchical multi-scale feature extraction on the image to be processed, and obtaining feature maps of different scales on the image to be processed through convolution kernels of different scales; The obtained feature maps of different scales are sent to the fusion layer to obtain the fused comprehensive feature map; Perform low-resolution object detection on the fused comprehensive feature map and output the target detection image; The fusion process includes: splicing feature maps of different scales and integrating multi-scale features through a convolution operation; A hierarchical attention mechanism is introduced based on the comprehensive feature map to adaptively adjust the feature importance of different regions in the comprehensive feature map; The spatial domain and frequency domain features are fused, and the comprehensive feature map is upsampled by interpolation method to expand the comprehensive feature map to the target resolution.

[0014] Preferably, the three-level identification strategy includes: Instruction scheduling based on the multi-stage pipeline of the MIPS instruction set architecture; Perform memory access based on a set number of memory accesses and a memory access pattern; Dynamically adjust the operating frequency and voltage of the CPU and neural network accelerator according to the current task load; Dynamically screening feature areas in the image to be processed whose importance meets the requirements for identification by a third identification algorithm; The third recognition algorithm comprises: Performing sparse processing on the image to be processed; Apply convolution kernels to the sparsely processed image to perform adaptive sparse convolution operations, dynamically skipping zero-value features during the convolution process; The convolutional image is passed through the pooling layer and the fully connected layer for feature extraction and classification; Output target detection image, including the location, category and confidence of the target.

[0015] According to a second aspect of an embodiment of the present application, a low-resolution object detection system based on heterogeneous computing is provided, including: A receiving module, used for receiving an image processing request and an image to be processed sent by a user; A determination module, used to determine the resolution level of the image to be processed and obtain the energy consumption level of the current system; A calling module is used to call the corresponding level of identification module according to the energy consumption level of the current system; An adjustment module, used to adjust the execution efficiency of the recognition module based on the called recognition module according to the resolution level of the image to be processed, as an execution module; An execution module, configured to perform low-resolution object detection on the image to be processed and output an object detection image; The levels of the identification modules include at least: a primary identification module, a secondary identification module and a tertiary identification module; Different levels of recognition modules use different recognition algorithms. The energy consumption and processing capacity of the first-level recognition module are greater than those of the second-level recognition module, which are greater than those of the third-level recognition module.

[0016] The technical solution provided by this application may have the following beneficial effects: The low-resolution object target detection method based on heterogeneous computing in the present application includes: receiving an image processing request sent by a user and an image to be processed; determining the resolution level of the image to be processed and obtaining the energy consumption level of the current system; calling a recognition strategy of the corresponding level according to the energy consumption level of the current system; adjusting the execution efficiency of the recognition strategy based on the called recognition strategy according to the resolution level of the image to be processed as the execution strategy; performing low-resolution object target detection on the image to be processed through the execution strategy, and outputting a target detection image; wherein the levels of the recognition strategy include at least: a first-level recognition strategy, a second-level recognition strategy and a third-level recognition strategy; the recognition algorithms adopted by the recognition strategies of each level are different, and the energy consumption and processing capability of the first-level recognition strategy are greater than those of the second-level recognition strategy and greater than those of the third-level recognition strategy.

[0017] In this technical solution, the energy consumption level of the current system is first obtained, and the corresponding level of recognition strategy is called according to the energy consumption level of the current system, so as to ensure that the called recognition strategy does not occupy a large amount of energy consumption of the current system. Then, according to the resolution level of the image to be processed, the execution efficiency of the recognition strategy is adjusted to further reduce the system energy consumption when performing target recognition. In addition, adjusting the execution efficiency of the recognition strategy according to the resolution level of the image to be processed can also ensure the processing efficiency of the image to be processed, thereby achieving a balance between efficiency and energy consumption.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 It is a flowchart of a low-resolution object detection method based on heterogeneous computing provided by an embodiment of the present application; Figure 2 It is a structural diagram of a low-resolution object detection system based on heterogeneous computing provided by an embodiment of the present application.

[0021] Figure numerals: receiving module-21; determining module-22; calling module-23; adjusting module-24; executing module-25; primary identification module-26; secondary identification module-27; tertiary identification module-28. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] Embodiment 1 Figure 1 is a flowchart of a low-resolution object detection method based on heterogeneous computing provided by an embodiment of the present application, with reference to Figure 1 , a low-resolution object detection method based on heterogeneous computing, comprising: S11: receiving an image processing request and an image to be processed sent by a user; S12: determining the resolution level of the image to be processed and obtaining the energy consumption level of the current system; S13: calling the corresponding level identification strategy according to the energy consumption level of the current system; It should be noted that the levels of identification strategies include at least: primary identification strategy, secondary identification strategy and tertiary identification strategy; The recognition algorithms adopted by the recognition strategies at different levels are different. The energy consumption and processing capacity of the first-level recognition strategy are greater than those of the second-level recognition strategy, which are greater than those of the third-level recognition strategy.

[0024] The first-level identification strategy is the high-energy consumption identification strategy, the second-level identification strategy is the energy consumption and efficiency balance identification strategy, and the third-level identification strategy is the low-energy consumption identification strategy.

[0025] In this embodiment, the energy consumption occupied by the current system during operation is used as the energy consumption level of the system. In this way, when the energy consumption level of the current system is low, the high-energy consumption identification strategy can be called; when the energy consumption level of the current system is normal, the energy consumption and efficiency balance identification strategy is called; when the energy consumption level of the current system is high, the low-energy consumption identification strategy is called to reduce the total energy consumption of the system.

[0026] S14: Based on the called recognition strategy, adjusting the execution efficiency of the recognition strategy according to the resolution level of the image to be processed as the execution strategy; It should be noted that the lower the image resolution level, the higher the processing power required, and the higher the system energy consumption during processing. Conversely, when the image resolution level is higher, the lower the processing power required, and the lower the system energy consumption during processing.

[0027] For example: The resolution level of the image to be processed is divided into 10 levels, with level 10 being the lowest and level 1 being the highest.

[0028] If the resolution level of the current image to be processed is level 1, the execution efficiency of the recognition strategy is adjusted to 10%. If the resolution level of the current image to be processed is level 10, the execution efficiency of the recognition strategy is adjusted to 100%, and so on.

[0029] S15: Perform low-resolution object detection on the image to be processed by executing the strategy, and output a target detection image.

[0030] In this technical solution, the energy consumption level of the current system is first obtained, and the corresponding level of recognition strategy is called according to the energy consumption level of the current system, so as to ensure that the called recognition strategy does not occupy a large amount of energy consumption of the current system. Then, according to the resolution level of the image to be processed, the execution efficiency of the recognition strategy is adjusted to further reduce the system energy consumption when performing target recognition. In addition, adjusting the execution efficiency of the recognition strategy according to the resolution level of the image to be processed can also ensure the processing efficiency of the image to be processed, thereby achieving a balance between efficiency and energy consumption.

[0031] Embodiment 2 It should be noted that the method also includes: Determine whether the image processing request contains the recognition strategy level required by the user; If the image processing request includes a recognition strategy level requested by the user, determining whether the recognition strategy level requested by the user exceeds the currently called recognition strategy level; If the identification strategy level requested by the user does not exceed the currently called identification strategy level, the current execution strategy is maintained; If the recognition strategy level required by the user exceeds the recognition strategy level currently called, determine whether the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold; If the execution efficiency corresponding to the current execution strategy does not exceed the set efficiency threshold level, the execution efficiency of the identification strategy is adjusted to the maximum value, and the execution strategy is updated; If the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold, the execution strategy is updated according to the identification strategy level required by the user.

[0032] In this embodiment, taking into account that the user may request the recognition policy level of his own, the recognition policy level required by the user is included in the image processing request, and it is necessary to determine whether the recognition policy level required by the user exceeds the recognition policy level currently called. If the recognition policy level required by the user does not exceed the recognition policy level currently called, it means that the recognition policy level currently called can meet the user's requirements, and the current execution policy is maintained.

[0033] If the identification strategy level required by the user exceeds the currently called identification strategy level, it means that the currently called identification strategy level fails to meet the user's requirements. At this time, the specific implementation plan needs to be determined according to the current situation: Determine whether the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold, which can be 50%. If the execution efficiency corresponding to the current execution strategy does not exceed the set efficiency threshold level, it means that the resolution of the current image to be processed is relatively high and is above the average line. In this way, the execution efficiency of the recognition strategy is adjusted to the maximum value, that is, the low-resolution object target detection is performed on the image to be processed with 100% execution efficiency of the current recognition strategy. In this way, user needs can be met to a certain extent without changing the current recognition strategy.

[0034] If the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold, it means that the resolution of the current image to be processed is low and below the average line. At this time, even if the execution efficiency of the recognition strategy is adjusted to the maximum value, it cannot be effectively improved. The execution strategy needs to be updated according to the recognition strategy level required by the user. For example, if the recognition strategy required by the user is a first-level recognition strategy, and the current execution strategy is 70% of the execution power of the second-level recognition strategy, the execution strategy needs to be updated to 70% of the execution power of the third-level recognition strategy.

[0035] Embodiment 3 It should be noted that the method also includes: If the identification strategy level required by the user exceeds the preset identification strategy level, determining whether the current system energy consumption level is lower than the preset energy consumption threshold; If the energy consumption level of the current system is not lower than the preset energy consumption threshold, determine whether the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level identification strategy; If the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level recognition strategy, then on the basis of the full efficiency execution of the first-level recognition strategy, the second-level recognition strategy is synchronously executed with full efficiency to perform low-resolution object detection on the processed image; If the current execution strategy is a first-level recognition strategy, but the corresponding execution efficiency is not the maximum value of the first-level recognition strategy, then on the basis of the full-efficiency execution of the first-level recognition strategy, the third-level recognition strategy is synchronously executed with full efficiency to perform low-resolution object detection on the image to be processed.

[0036] In this embodiment, it is considered that the recognition strategy level required by the user may be too high, and multiple recognition strategies need to be run simultaneously to recognize the image to be processed. In this case, it is necessary to first consider whether the system supports the energy consumption of multiple recognition strategies running simultaneously.

[0037] First, determine whether the energy consumption level of the current system is lower than the preset energy consumption threshold. If the energy consumption level of the current system is not lower than the preset energy consumption threshold, it indicates that the energy consumption level of the current system is high and it is difficult to support the simultaneous operation of multiple identification strategies.

[0038] Since the recognition strategy level required by the user is very high, the current image to be processed can only be processed by the first-level recognition strategy supplemented by other recognition strategies.

[0039] At this time, it is determined whether the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level recognition strategy. If the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level recognition strategy, it means that the resolution of the current image to be processed is the lowest level, and two high-processing capability strategies, the first-level recognition strategy and the second-level recognition strategy, are needed to process the current image to be processed. At this time, based on the full efficiency execution of the first-level recognition strategy, the second-level recognition strategy is synchronously executed with full efficiency to perform low-resolution object detection on the image to be processed.

[0040] If the execution efficiency corresponding to the current execution strategy is not the maximum value of the first-level recognition strategy, it means that the resolution of the current image to be processed is not the lowest level. At this time, the first-level recognition strategy has been called, and there is no need to call the second-level recognition strategy to increase system energy consumption. Only the first-level recognition strategy can be used.

[0041] It should be noted that the method also includes: If the current system energy consumption level is lower than the preset energy consumption threshold, all recognition strategies are synchronously executed at full efficiency to perform low-resolution object detection on the image to be processed.

[0042] If the current system energy consumption level is lower than the preset energy consumption threshold, it means that the system energy consumption level is low and can support multiple recognition strategies to run simultaneously. At this time, all recognition strategies are directly and synchronously executed at full efficiency to perform low-resolution object detection on the processed images, which can better meet user needs.

[0043] The technical solution in this embodiment can achieve a compromise execution strategy between meeting user needs, reducing system energy consumption and improving processing efficiency when the user has a customized recognition strategy requirement.

[0044] Furthermore, the method also includes: Assign weights to each identification strategy; According to the weights assigned to each recognition strategy, the target detection results of all recognition strategies are fused and the consistency detection results are output.

[0045] Since in this embodiment, target detection is performed on the image to be processed using multiple recognition strategies, multiple target detection results are generated.

[0046] In this embodiment, each recognition strategy needs to be assigned a weight, such as a weight of 0.5 for the first-level recognition strategy, a weight of 0.3 for the second-level recognition strategy, and a weight of 0.2 for the third-level recognition strategy. According to the weights assigned to each recognition strategy, the target detection results of all recognition strategies are fused to output the consistency detection results.

[0047] Embodiment 4 It should be noted that the method also includes: Obtaining the parameters of the current display system, and determining the display image format according to the parameters of the current display system; Convert the target detection image into a display image format; The target detection image in the display image format is enhanced by histogram equalization and noise is filtered out by Gaussian filtering; Determine the resolution based on the parameters of the current display system; The resolution of the processed target detection image is adjusted according to the resolution of the current display system by bilinear interpolation; The target detection image with adjusted resolution is output through the current display system.

[0048] In specific practice, the display system is based on augmented reality technology and uses advanced array optical waveguide and LCOS projection technology to present the picture in front of the eyes with high precision while maintaining the user's clear view of the real scene. As an effective auxiliary terminal, the display system is suitable for diverse scenarios that require high-precision recognition and display.

[0049] In this technical solution, the image format is first converted to the standard display format of the current display system (such as RGB) to ensure that it is compatible with the display requirements of the LCOS projection system. Then, the contrast and brightness are improved through enhancement processing such as histogram equalization, and Gaussian filtering is used to filter out noise, thereby achieving a more delicate and smooth display effect. In order to adapt to the resolution of the display system, bilinear interpolation is used to scale the image to ensure that the image projected onto the optical waveguide array has the best display effect.

[0050] The display system is responsible for transmitting the processed image data to the augmented reality display device to ensure precise matching with the array optical waveguide and LCOS projection optical machine. First, the image data is converted into a signal format suitable for LCOS projection (such as HDMI or MIPI) through a signal conversion chip. According to the characteristics of the display device, set the display parameters such as resolution, brightness, and contrast to ensure the clarity and color saturation of the picture. The display driver projects the optimized image content onto the optical waveguide lens to form a virtual picture in the user's field of view, while allowing perspective of the real world, enabling users to superimpose virtual information on real scenes, thereby enhancing perception and understanding of the environment.

[0051] Embodiment 5 In this embodiment, a specific identification strategy is described.

[0052] The first-level identification strategy is suitable for high-performance usage scenarios, such as laboratories, computer rooms, data centers, etc.

[0053] The primary identification strategy includes: Based on the parallel computing capability and high-bandwidth memory access capability of GPU, low-resolution object detection is performed on the processed image through the first recognition algorithm; The first recognition algorithm includes: The global dependencies of the image to be processed are captured through the self-attention mechanism of the Transformer encoder; According to the global dependency of the image to be processed, noise is gradually added to the image to be processed. After T steps, a completely randomized feature map is obtained. Step by step, reduce the noise in the completely randomized feature map, and after T steps, restore the image to be processed; Extract features at different scales from the image to be processed after adding noise and reverse denoising, and fuse the extracted features; different weights are assigned to each scale; Perform low-resolution object detection on the image after feature fusion and output the target detection image.

[0054] The high-performance GPU is one of the core components of the first-level recognition strategy, with powerful parallel computing capabilities and high-bandwidth memory access capabilities. The high-performance GPU accelerates the computing tasks of neural networks through hardware, especially the forward propagation process of convolutional neural networks (CNNs). To further improve recognition accuracy and processing speed, the high-performance GPU is combined with a deeply customized neural network accelerator to optimize key computing tasks through hardware accelerators. The deeply customized neural network accelerator design includes multiple computing units and storage units, and achieves efficient data transmission and processing through on-chip memory and inter-chip communication mechanisms. To further optimize power consumption and performance, the accelerator uses a transformer-based diffusion model (TBDiff), which is specially optimized for low-resolution images.

[0055] The first recognition algorithm combines the Transformer model and the diffusion model to improve the model's expressiveness and processing speed, and is especially optimized for the image to be processed. Assume that the input image to be processed is , where H and W represent the height and width of the image to be processed, respectively, and C represents the number of channels. , where k represents the size of the convolution kernel and F represents the number of output channels. The traditional convolution operation can be expressed as:

[0056] The TBDiff algorithm combines the Transformer model and the diffusion model to improve the model's expressiveness and processing speed, and is especially optimized for low-resolution images. Specifically, the TBDiff algorithm is divided into two main steps: Transformer encoding and diffusion model generation.

[0057] 1. Transformer encoding: The Transformer encoder captures the global dependencies of the input feature map through the self-attention mechanism, which is particularly suitable for processing the details in low-resolution images. Assume that the input feature map X is expanded into N vectors ,in Each vector The output of the Transformer encoder can be expressed as:

[0058] Specifically, the self-attention mechanism of the Transformer encoder can be expressed as:

[0059]

[0060] in, , , is a learnable weight matrix, is the feature dimension. After the multi-head self-attention mechanism, the output Z can be expressed as:

[0061]

[0062] Where h is the number of heads, is the final output weight matrix.

[0063] 2. The diffusion model generates high-quality feature maps by gradually adding noise and reverse denoising, which is particularly suitable for processing detailed information in low-resolution images. Assume that the initial feature map , the diffusion process can be expressed as:

[0064] in, is the noise addition factor, is the random noise of standard normal distribution. The inverse denoising process can be expressed as:

[0065] in, It is the noise predicted by the neural network. The whole diffusion model process can be described as: 1. Forward diffusion process: From the initial feature map Start by adding noise gradually.

[0066] The diffusion process of each step can be expressed as:

[0067] After T steps, a completely randomized feature map is obtained. .

[0068] 2. Backward denoising process: From a completely randomized feature map Start by reducing the noise gradually.

[0069] The denoising process of each step can be expressed as:

[0070] After T steps, the initial feature map is restored .

[0071] In order to further optimize the processing of low-resolution images, the TBDiff algorithm introduces a multi-scale feature fusion mechanism. Specifically, the multi-scale feature fusion mechanism extracts features at different scales and fuses these features to improve the model's recognition ability for low-resolution images. Assume that the multi-scale feature maps are , , , multi-scale feature fusion can be expressed as:

[0072] in, is the weight of feature maps of different scales, Is the first The feature map of each scale is upsampled to the initial feature map In this way, the multi-scale feature fusion mechanism can make full use of information at different scales and improve the model's recognition ability for low-resolution images.

[0073] In order to further optimize the processing of low-resolution images, the TBDiff algorithm introduces a multi-scale feature fusion mechanism. Specifically, the multi-scale feature fusion mechanism extracts features at different scales and fuses these features to improve the model's recognition ability for low-resolution images. Assume that the multi-scale feature maps are , the multi-scale feature fusion function can be expressed as:

[0074] in, It is The weights on each scale can be obtained through training.

[0075] Final feature map Further processing is performed through a high-performance GPU and a deeply customized neural network accelerator to generate high-precision target detection results.

[0076] Secondary identification strategies include: Based on ARM's multi-core parallel structure, the frequency can be dynamically adjusted according to different loads; Performing low-resolution object detection on the processed image by a second recognition algorithm; The second recognition algorithm includes: Perform hierarchical multi-scale feature extraction on the image to be processed, and obtain feature maps of different scales on the image to be processed through convolution kernels of different scales; The obtained feature maps of different scales are sent to the fusion layer to obtain the fused comprehensive feature map; Perform low-resolution object detection on the fused comprehensive feature map and output the target detection image; The fusion process includes: splicing feature maps of different scales and integrating multi-scale features through a convolution operation; A hierarchical attention mechanism is introduced based on the comprehensive feature map to adaptively adjust the feature importance of different regions in the comprehensive feature map. The spatial domain and frequency domain features are fused, and the comprehensive feature map is upsampled by interpolation method to expand the comprehensive feature map to the target resolution.

[0077] The secondary recognition strategy is designed to meet the needs of smart devices to efficiently perform target detection tasks under low power consumption conditions. With the increasing requirements for long-term, high-performance operation in application scenarios such as home security monitoring systems and small drones, there is an urgent need for a module that has both high-performance computing capabilities and strict power consumption control to achieve long-term stable operation of the device and high-quality target detection. The secondary recognition strategy combines an efficient processor based on the ARM architecture with an optimized neural network accelerator, and incorporates an adaptive multi-scale super-resolution algorithm (HAMSR) at the algorithm level to achieve efficient reconstruction and target detection of low-resolution images. The HAMSR algorithm uses technical means such as hierarchical attention, multi-scale feature extraction, and spatial frequency domain fusion to ensure that the module maintains high accuracy and high real-time performance when processing low-resolution images, while significantly reducing power consumption.

[0078] The hardware part of the secondary recognition strategy is based on an efficient processor based on the ARM architecture and a specially designed neural network accelerator. The processor adopts a multi-core parallel structure, which can dynamically adjust the frequency according to different loads to achieve a dynamic balance between performance and power consumption. By combining the hardware optimization design of the neural network accelerator, the core calculations of the image feature extraction and target detection processes can be accelerated. This hardware combination structure greatly improves the overall computing power and provides basic support for the efficient operation of the HAMSR algorithm.

[0079] At the algorithm level, the design focus of the HAMSR algorithm (the second recognition algorithm) is on multi-scale adaptive feature extraction and spatial frequency domain feature fusion. The HAMSR algorithm first processes the input image to be processed. Perform hierarchical multi-scale feature extraction. Assume the image size is , the number of channels is C , extract features through three different scales. Specifically, the convolution kernel size is Three sets of convolution operations are used to extract feature maps of different scales. , in order to capture the detailed features in the image. The convolution operation at each scale is defined as:

[0080] in, and Represent the convolution kernel weight and bias respectively, is the convolution kernel size, is the stride, is the filling size, Through convolution kernels of different scales, the module obtains feature maps of different resolutions on the same input image, ensuring that detail features and global features are effectively combined.

[0081] Multi-scale feature maps In the HAMSR algorithm, it is sent to the fusion layer to obtain the integrated comprehensive feature map In order to preserve multi-scale information during the fusion process, the HAMSR algorithm first concatenates the feature maps of each scale, and then uses a convolution operation to integrate the multi-scale features so that it maintains a high information density in the next processing step. The fusion operation expression is:

[0082] in, represents the concatenation operation of the feature map, and is the convolution kernel weight and bias of the fusion layer, k is the convolution kernel size, s is the stride, p This operation ensures that the fused feature map contains important information of each scale.

[0083] To further improve the reconstruction accuracy of low-resolution images, the HAMSR algorithm introduces a hierarchical attention mechanism based on the comprehensive feature map F_{\text{fuse}} to adaptively adjust the importance of features in different regions. The hierarchical attention mechanism generates an attention weight matrix by combining a convolution and an activation function. A , and apply it to the element-by-element multiplication of the comprehensive feature map to highlight the key areas in the image. The expression of the hierarchical attention mechanism is:

[0084]

[0085] in, represents the activation function, and are the convolution kernel weights and biases, is element-by-element multiplication. The attention mechanism enables the module to adaptively identify key areas of the image, effectively improving the local refinement of image reconstruction.

[0086] In the feature map fusion process, the features of the spatial domain and the frequency domain are further fused to enhance the expression of image edges and details when processing low-resolution images. The upsampling layer expands the image to the high-resolution image size. The upsampling operation is implemented through interpolation methods, such as bilinear interpolation or sub-pixel convolution, to make the image size consistent with the high-resolution target image. Feature map after upsampling The expression is:

[0087] in, r is the upsampling ratio, usually r =2, which means the image size is doubled. After upsampling, the feature map is passed to the final reconstruction layer to generate the final high-resolution image. :

[0088] in, and are the convolution kernel weights and biases. This high-resolution image has sufficient details and high-quality resolution to provide accurate data support for the target detection module.

[0089] High-resolution images After generation, it will be input into the target detection module to complete the target detection and recognition tasks. The high-quality images provided by the HAMSR algorithm can still achieve high-precision target positioning and classification in the case of low-resolution input, thereby ensuring detection accuracy under low power consumption conditions.

[0090] The three-level identification strategy includes: Instruction scheduling based on the multi-stage pipeline of the MIPS instruction set architecture; Perform memory access based on a set number of memory accesses and a memory access pattern; Dynamically adjust the operating frequency and voltage of the CPU and neural network accelerator according to the current task load; Dynamically screening feature areas in the image to be processed whose importance meets the requirements for identification through a third identification algorithm; The third recognition algorithm includes: Perform sparse processing on the image to be processed; Apply convolution kernels to the sparsely processed image to perform adaptive sparse convolution operations, dynamically skipping zero-value features during the convolution process; The convolutional image is passed through the pooling layer and the fully connected layer for feature extraction and classification; Output target detection image, including the location, category and confidence of the target.

[0091] The three-level recognition strategy is designed for extremely low-power usage scenarios, such as long-term industrial monitoring, automatic recognition of robots and unmanned ships, and deep space exploration. The three-level recognition strategy achieves efficient target detection and recognition under low-power conditions by combining a five-stage pipeline low-power CPU based on the MIPS instruction set architecture and a deeply customized general-purpose neural network accelerator.

[0092] The hardware architecture of the three-level recognition strategy is highly integrated, mainly including a five-stage pipeline low-power CPU based on the MIPS instruction set architecture and a deeply customized general neural network accelerator. The five-stage pipeline low-power CPU reduces unnecessary energy consumption by optimizing the instruction execution process while maintaining high computing efficiency. The general neural network accelerator accelerates key computing tasks in the neural network reasoning process through hardware-level optimization, further reducing power consumption.

[0093] In order to achieve efficient target detection under low power consumption conditions, the three-level recognition strategy introduces an innovative lightweight neural network algorithm (ASCN) as the third recognition algorithm. The ASCN algorithm reduces redundant calculations in convolution operations through adaptive sparse convolution technology, thereby significantly reducing computational complexity and power consumption while maintaining high detection accuracy.

[0094] The core idea of ​​the ASCN algorithm is to dynamically select important feature map regions for calculation in the convolution operation to avoid comprehensive convolution operations on the entire feature map. Specifically, the ASCN algorithm first performs sparse processing on the image to be processed. Input sparse image ,in H and W Represent the height and width of the feature map respectively, C Indicates the number of channels. Through a sparse layer S Process the input feature map to generate a sparse feature map . Sparse layer S The specific implementation can be a threshold-based filter, that is:

[0095] in, is a preset threshold that determines which feature values ​​are considered important.

[0096] In sparse feature maps Apply convolution kernel on ,in k is the size of the convolution kernel, F is the number of channels of the output feature map. The adaptive sparse convolution operation can be expressed by the following formula:

[0097] To further reduce the amount of computation, zero-value features can be dynamically skipped during the convolution process, i.e.:

[0098] in, is the set of indices of all non-zero eigenvalues.

[0099] Feature map after convolution Y It is passed to the subsequent pooling layer and fully connected layer for further feature extraction and classification. The final target detection result is output, including the location, category and confidence of the target.

[0100] Embodiment 6 A low-resolution object detection system based on heterogeneous computing, referring to Figure 2 ,include: The receiving module 21 is used to receive an image processing request and an image to be processed sent by a user; A determination module 22, used to determine the resolution level of the image to be processed and obtain the energy consumption level of the current system; A calling module 23 is used to call a recognition module of a corresponding level according to the energy consumption level of the current system; An adjustment module 24, used to adjust the execution efficiency of the recognition module based on the called recognition module and according to the resolution level of the image to be processed, as an execution module; An execution module 25 is used to perform low-resolution object detection on the image to be processed and output an object detection image; The levels of the recognition modules include at least: a primary recognition module 26, a secondary recognition module 27 and a tertiary recognition module 28; Different levels of recognition modules use different recognition algorithms. The energy consumption and processing capacity of the first-level recognition module are greater than those of the second-level recognition module, which are greater than those of the third-level recognition module.

[0101] In this system, the energy consumption level of the current system is first obtained, and the corresponding level of recognition module is called according to the energy consumption level of the current system, so as to ensure that the called recognition module does not occupy a large amount of energy consumption of the current system. Then, according to the resolution level of the image to be processed, the execution efficiency of the recognition module is adjusted to further reduce the system energy consumption when performing target recognition. In addition, adjusting the execution efficiency of the recognition strategy according to the resolution level of the image to be processed can also ensure the processing efficiency of the image to be processed, thereby achieving a balance between efficiency and energy consumption.

[0102] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0103] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0104] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0105] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0106] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0108] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0110] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A low-resolution object detection method based on heterogeneous computing, characterized in that: include: Receive image processing requests and images to be processed sent by users; Determine the resolution level of the image to be processed, and obtain the energy consumption level of the current system; Call the corresponding level identification strategy according to the energy consumption level of the current system; Based on the called recognition strategy, adjusting the execution efficiency of the recognition strategy according to the resolution level of the image to be processed as the execution strategy; Performing low-resolution object detection on the image to be processed by the execution strategy, and outputting an object detection image; The levels of the identification strategy include at least: a primary identification strategy, a secondary identification strategy, and a tertiary identification strategy; The recognition algorithms adopted by the recognition strategies of different levels are different. The energy consumption and processing capability of the first-level recognition strategy are greater than those of the second-level recognition strategy and greater than those of the third-level recognition strategy.

2. The method according to claim 1, characterized in that The method further comprises: Determining whether the image processing request includes a recognition strategy level required by a user; If the image processing request includes a recognition strategy level required by the user, determining whether the recognition strategy level required by the user exceeds the currently called recognition strategy level; If the identification strategy level requested by the user does not exceed the currently called identification strategy level, the current execution strategy is maintained; If the recognition strategy level required by the user exceeds the recognition strategy level currently called, determine whether the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold; If the execution efficiency corresponding to the current execution strategy does not exceed the set efficiency threshold level, the execution efficiency of the identification strategy is adjusted to the maximum value, and the execution strategy is updated; If the execution efficiency corresponding to the current execution strategy exceeds the set efficiency threshold, the execution strategy is updated according to the identification strategy level required by the user.

3. The method according to claim 2, characterized in that The method further comprises: If the identification strategy level required by the user exceeds the preset identification strategy level, determining whether the current system energy consumption level is lower than the preset energy consumption threshold; If the energy consumption level of the current system is not lower than the preset energy consumption threshold, determine whether the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level identification strategy; If the execution efficiency corresponding to the current execution strategy is the maximum value of the first-level recognition strategy, then based on the full-efficiency execution of the first-level recognition strategy, the second-level recognition strategy is synchronously executed with full efficiency to perform low-resolution object target detection on the image to be processed; If the current execution strategy is a first-level recognition strategy, but the corresponding execution efficiency is not the maximum value of the first-level recognition strategy, then on the basis of the full-efficiency execution of the first-level recognition strategy, the third-level recognition strategy is synchronously executed with full efficiency to perform low-resolution object target detection on the image to be processed.

4. The method according to claim 3, characterized in that The method further comprises: If the energy consumption level of the current system is lower than the preset energy consumption threshold, all recognition strategies are synchronously executed with full efficiency to perform low-resolution object detection on the image to be processed.

5. The method according to any one of claims 3-4, characterized in that: The method further comprises: Assign weights to each identification strategy; According to the weights assigned to each recognition strategy, the target detection results of all recognition strategies are fused and the consistency detection results are output.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining the parameters of the current display system, and determining the display image format according to the parameters of the current display system; Converting the target detection image into the display image format; The target detection image in the display image format is enhanced by histogram equalization and noise is filtered out by Gaussian filtering; Determine the resolution based on the parameters of the current display system; The resolution of the processed target detection image is adjusted according to the resolution of the current display system by bilinear interpolation; The target detection image with adjusted resolution is output through the current display system.

7. The method according to claim 1, characterized in that The primary identification strategy includes: Based on the parallel computing capability and high-bandwidth memory access capability of the GPU, low-resolution object target detection is performed on the image to be processed by a first recognition algorithm; The first recognition algorithm comprises: Capturing the global dependencies of the image to be processed through the self-attention mechanism of the Transformer encoder; According to the global dependency of the image to be processed, noise is gradually added to the image to be processed. After T steps, a completely randomized feature map is obtained. Step by step, reduce the noise in the completely randomized feature map, and after T steps, restore the image to be processed; Extract features at different scales from the image to be processed after adding noise and reverse denoising, and fuse the extracted features; different weights are assigned to each scale; Perform low-resolution object detection on the image after feature fusion and output the target detection image.

8. The method according to claim 1, characterized in that The secondary identification strategy includes: Based on ARM's multi-core parallel structure, the frequency can be dynamically adjusted according to different loads; Performing low-resolution object detection on the image to be processed by a second recognition algorithm; The second recognition algorithm comprises: Performing hierarchical multi-scale feature extraction on the image to be processed, and obtaining feature maps of different scales on the image to be processed through convolution kernels of different scales; The obtained feature maps of different scales are sent to the fusion layer to obtain the fused comprehensive feature map; Perform low-resolution object detection on the fused comprehensive feature map and output the target detection image; The fusion process includes: splicing feature maps of different scales and integrating multi-scale features through a convolution operation; A hierarchical attention mechanism is introduced based on the comprehensive feature map to adaptively adjust the feature importance of different regions in the comprehensive feature map; The spatial domain and frequency domain features are fused, and the comprehensive feature map is upsampled by interpolation method to expand the comprehensive feature map to the target resolution.

9. The method according to claim 1, characterized in that: The three-level identification strategy includes: Instruction scheduling based on the multi-stage pipeline of the MIPS instruction set architecture; Perform memory access based on a set number of memory accesses and a memory access pattern; Dynamically adjust the operating frequency and voltage of the CPU and neural network accelerator according to the current task load; Dynamically screening feature areas in the image to be processed whose importance meets the requirements for identification by a third identification algorithm; The third recognition algorithm comprises: Performing sparse processing on the image to be processed; Apply convolution kernels to the sparsely processed image to perform adaptive sparse convolution operations, dynamically skipping zero-value features during the convolution process; The convolutional image is passed through the pooling layer and the fully connected layer for feature extraction and classification; Output target detection image, including the location, category and confidence of the target.

10. A low-resolution object detection system based on heterogeneous computing, characterized in that: include: A receiving module, used for receiving an image processing request and an image to be processed sent by a user; A determination module, used to determine the resolution level of the image to be processed and obtain the energy consumption level of the current system; A calling module is used to call the corresponding level of identification module according to the energy consumption level of the current system; An adjustment module, used to adjust the execution efficiency of the recognition module based on the called recognition module according to the resolution level of the image to be processed, as an execution module; An execution module, configured to perform low-resolution object detection on the image to be processed and output an object detection image; The levels of the identification modules include at least: a primary identification module, a secondary identification module and a tertiary identification module; Different levels of recognition modules use different recognition algorithms. The energy consumption and processing capacity of the first-level recognition module are greater than those of the second-level recognition module, which are greater than those of the third-level recognition module.

Images