Edge-end image recognition chip acceleration method and system based on multi-scale fusion

Abstract

The application discloses an edge image recognition chip acceleration method and system based on multi-scale fusion. The method first collects images and standardizes them, and outputs three layers of multi-scale features through a fusion convolution network; redundant background features are filtered and compressed through scale information entropy block screening; three types of feature partition caching are realized relying on double-buffered ping-pong read-write; a bidirectional cross-scale branch is built to solve a cross-correlation matrix and generate dynamic fusion weights, and multiple operators are arranged in parallel, and the target quantity is adaptively scheduled to power; finally, the target category and coordinates are output through a double-branch recognition head. The system cooperates with six interconnected modules, reduces data overhead while ensuring recognition accuracy, adapts to edge chip power and storage limited scenes, and realizes low latency and low power target detection acceleration. The method relates to the field of edge neural network acceleration technology, and solves the technical problems of feature data redundancy, fixed fusion weight, and lack of differentiation in storage and power scheduling in the prior art.

Description

Technical Field

[0001] This invention relates to the field of edge neural network acceleration technology, and more specifically, to an edge image recognition chip acceleration method and system based on multi-scale fusion. Background Technology

[0002] With the rapid development of intelligent monitoring, vehicle sensing, and embedded vision devices, a large number of image recognition tasks need to be deployed on edge hardware chips. However, edge chips have limited computing power and on-chip storage resources, and traditional target recognition solutions based on convolutional neural networks face problems such as high inference latency, large bandwidth overhead, and difficulty in controlling power consumption.

[0003] In object detection tasks, multi-scale feature fusion is a core method to improve the accuracy of recognizing targets of different sizes, and feature pyramid structures such as FPN and BiFPN are widely used. These schemes directly involve the shallow, mid-level, and deep features output by the network in the fusion operation. Shallow feature maps have high resolution and contain a large amount of redundant background information; direct transmission and computation without filtering consumes a large amount of on-chip cache and computing resources, increasing the data read / write burden on edge chips. Furthermore, most existing fusion methods use fixed fusion weights after training, which cannot adaptively adjust the contribution of features at each scale according to the distribution of targets within the image. This makes it difficult to simultaneously consider the detailed features of small targets and the semantic features of large targets, resulting in a bottleneck in recognition accuracy.

[0004] Furthermore, existing edge inference processes lack a layered and differentiated data caching strategy, storing all types of features uniformly, with high-value target features and redundant background features occupying equal storage resources. Most computing architectures adopt a serial execution mode, with feature selection and feature fusion being processed sequentially, resulting in numerous waiting bubbles in the hardware pipeline. Moreover, computing resources are allocated statically, making it impossible to dynamically adjust computing power based on the number of effective targets in the image. This leads to wasted computing power in sparse target scenarios and insufficient computing power in dense target scenarios, making it difficult to achieve a balance between recognition accuracy, inference latency, and hardware power consumption.

[0005] Therefore, there is an urgent need for an image recognition acceleration method that can reduce feature transmission redundancy, achieve adaptive multi-scale feature fusion, and adapt to the computing power characteristics of edge image recognition chips. Summary of the Invention

[0006] The main objective of this invention is to provide an edge image recognition chip acceleration method and system based on multi-scale fusion, so as to at least solve the technical problems of feature data redundancy, fixed fusion weights, and lack of differentiation in storage and computing power scheduling in the prior art. While ensuring recognition accuracy, it reduces on-chip data overhead and achieves low latency and low power consumption accelerated inference for edge image recognition.

[0007] To achieve the above objectives, an edge image recognition chip acceleration method and system based on multi-scale fusion is provided.

[0008] In a first aspect, the present invention provides an acceleration method for edge image recognition chips based on multi-scale fusion, the acceleration method comprising:

[0009] The original image data of the target area is acquired using an image sensor. The original image data is normalized and scaled to obtain standard image data. A fusion convolutional recognition network is constructed. The standard image data is input into the fusion convolutional recognition network for forward inference to obtain multi-scale original features. The multi-scale original features include a first feature map, a second feature map, and a third feature map.

[0010] A preset size threshold and entropy threshold are used to divide the first feature map and the second feature map into multiple non-overlapping feature blocks according to the size threshold. The scale information entropy of each non-overlapping feature block is calculated. The multiple scale information entropies are compared with the entropy threshold to obtain effective target blocks, smooth background blocks and blank invalid blocks. The smooth background blocks are sampled and compressed to obtain compressed background blocks, and blank invalid blocks are discarded. The effective target blocks and compressed background blocks in the first feature map are recombined to obtain the filtered first feature. The effective target blocks and compressed background blocks in the second feature map are recombined to obtain the filtered second feature.

[0011] The double-buffered ping-pong read-write mechanism is used to write the effective target blocks in the first and second filtered features into the feature micro-buffer, write the compressed background blocks in the first and second filtered features into a temporary FIFO, and store the third feature map into the semantic buffer. The double-buffered ping-pong read-write mechanism can perform data writing and data reading operations in parallel.

[0012] In the fused convolutional recognition network, a bidirectional cross-scale interactive branch is constructed. The effective target blocks of the first and second filtered features are retrieved from the feature micro-buffer, and the compressed background blocks of the first and second filtered features are retrieved from the temporary FIFO. The first and second feature maps are then concatenated and recombined according to their spatial positions to obtain the complete first and second filtered features. The third feature map is then read from the semantic buffer. The scale cross-correlation matrix of the first, second, and third filtered features is solved using the bidirectional cross-scale interactive branch, and dynamic fusion weights are generated in real time.

[0013] Based on dynamic fusion weights, the first feature, the second feature, and the third feature map after screening are weighted and fused to obtain a fused feature map. When calculating the scale information entropy, the scale cross-correlation matrix, and the fused feature map, a multi-operator parallel hybrid computing method is used for synchronous operation, and computing resources are dynamically allocated according to the number of effective target blocks.

[0014] The fused feature map is input into the recognition head of the fused convolutional recognition network. The recognition head is used to classify and detect the target using the fused feature map to obtain the target category and target coordinates.

[0015] Specifically, the raw image data of the target area is acquired using an image sensor, and the raw image data is normalized and scaled to obtain standard image data, including:

[0016] Preset resolution and frame rate, and use an image sensor to acquire raw image data of the target area based on the resolution and frame rate;

[0017] The pixel values ​​of the original image data are uniformly mapped to the standard range of [0,1] to obtain normalized image data;

[0018] A standard size is preset, and the normalized image data is scaled to the standard size using a bilinear interpolation algorithm while preserving edge details and texture features to obtain standard image data.

[0019] Specifically, standard image data is input into a fusion convolutional recognition network for forward inference to obtain multi-scale raw features, including:

[0020] First, the first feature map in the standard image data is extracted using the ordinary convolutional layer in the fused convolutional recognition network.

[0021] Secondly, the separable convolutional layers in the fused convolutional recognition network are used to extract the second feature map from the standard image data;

[0022] Finally, the dilated convolutional layer in the fused convolutional recognition network is used to extract the third feature map from the standard image data.

[0023] Specifically, the first feature map and the second feature map are divided into multiple non-overlapping feature blocks according to a size threshold, and the scale information entropy of each non-overlapping feature block is calculated, including:

[0024] A preset size threshold is used to divide the first feature map and the second feature map into multiple non-overlapping feature blocks based on the feature map resolution and the size threshold of the first feature map and the second feature map.

[0025] The scale information entropy of each non-overlapping feature block is calculated by traversing multiple non-overlapping feature blocks and using a channel-by-channel independent calculation strategy.

[0026] Specifically, multiple scale information entropy and entropy thresholds are compared to obtain effective target blocks, smooth background blocks, and blank invalid blocks. Sampling and compression of the smooth background blocks yields compressed background blocks, while blank invalid blocks are discarded. The effective target blocks and compressed background blocks in the first feature map are recombined to obtain the filtered first feature. The effective target blocks and compressed background blocks in the second feature map are recombined to obtain the filtered second feature, including:

[0027] A first entropy threshold and a second entropy threshold are preset. Non-overlapping feature blocks with scale information entropy greater than or equal to the first entropy threshold are determined as valid target blocks; non-overlapping feature blocks with scale information entropy greater than or equal to the second entropy threshold and less than the first entropy threshold are determined as smooth background blocks; and non-overlapping feature blocks with scale information entropy less than the second entropy threshold are determined as blank invalid blocks.

[0028] A mean-downsampling compression algorithm is used to sample and compress the smooth background block to obtain a compressed background block, while discarding blank and invalid blocks.

[0029] Based on the spatial relationship between the first feature map and the second feature map, the effective target block and the compressed background block in the first feature map are recombined to obtain the filtered first feature, and the effective target block and the compressed background block in the second feature map are recombined to obtain the filtered second feature. The filtered first feature and the filtered second feature are consistent with the feature size and spatial structure of the first feature map and the second feature map, respectively.

[0030] Specifically, a double-buffered ping-pong read-write mechanism is used to write the effective target blocks from the first and second filtered features into the feature micro-buffer, write the compressed background blocks from the first and second filtered features into a temporary FIFO, and store the third feature map in the semantic buffer, including:

[0031] A double-buffered ping-pong read-write mechanism is used to store the effective target blocks from the first and second filtered features into a feature micro-cache with small capacity and fast read-write speed.

[0032] A double-buffered ping-pong read-write mechanism is used to write the compressed background block from the first and second filtered features into a temporary FIFO.

[0033] The third feature map is stored in the semantic cache using a double-buffered ping-pong read-write mechanism;

[0034] The dual-buffered ping-pong read / write mechanism includes a first buffer unit and a second buffer unit, which alternately perform data write and data read operations.

[0035] Among them, the feature micro-cache, temporary FIFO and semantic cache are read and written independently and do not interfere with each other.

[0036] Specifically, a bidirectional cross-scale interaction branch is constructed in the fused convolutional recognition network. Valid target blocks of the first and second filtered features are retrieved from the feature micro-buffer, and compressed background blocks of the first and second filtered features are retrieved from a temporary FIFO. These blocks are then concatenated and recombined according to their spatial positions to obtain complete first and second filtered features. Next, the third feature map is read from the semantic buffer, and the scale cross-correlation matrix of the first, second, and third filtered features is solved using the bidirectional cross-scale interaction branch. Dynamic fusion weights are generated in real time, including:

[0037] Construct a bidirectional cross-scale interaction branch, which includes a top-down semantic transmission branch and a bottom-up detail feedback branch. The semantic transmission branch transmits the global semantic information of the third feature map upwards, and the detail feedback branch transmits the filtered first feature and the filtered second feature downwards.

[0038] The effective target blocks of the first and second filtered features are retrieved from the feature micro-cache, respectively. The compressed background blocks of the first and second filtered features are retrieved from the temporary FIFO. The first and second feature maps are then spliced ​​and recombined according to their spatial positions to obtain the complete first and second filtered features. The third feature map is then read from the semantic cache.

[0039] The spatial and channel correlations of the first, second, and third feature maps after filtering are solved by matrix operations. A scale cross-correlation matrix is ​​constructed based on the spatial and channel correlations. Dynamic fusion weights are adaptively allocated based on the correlation degree values ​​of the scale cross-correlation matrix.

[0040] Specifically, when calculating the scale information entropy, scale cross-correlation matrix, and fused feature map, a multi-operator parallel hybrid computing approach is used for synchronous computation. Computational resources are dynamically allocated based on the number of effective target blocks, including:

[0041] A preset quantity threshold is used to determine the number of valid target blocks obtained after the inference of the previous frame, which is then used as the basis for computing power scheduling in the current frame.

[0042] In the single-frame inference pipeline, the calculation of scale information entropy and the reorganization of feature selection are completed first, and then the solution of scale cross-correlation matrix and the calculation of fused feature map are performed.

[0043] When the number of valid target blocks in the previous frame is greater than or equal to the number threshold, it is determined to be a target-dense scene. In the current frame, more computing power is reserved and parallel threads are preferentially allocated to the calculation stage of scale cross-correlation matrix and fusion feature map.

[0044] When the number of valid target blocks in the previous frame is less than the number threshold, it is determined to be a sparse target scene. In the current frame, more computing power will be allocated to the calculation and feature selection stage of scale information entropy block. Among them, each stage adopts a multi-operator parallel hybrid operation method, and different stages are executed sequentially according to data dependencies.

[0045] Specifically, the fused feature map is input into the recognition head of the fused convolutional recognition network. The recognition head then performs classification and target detection on the fused feature map to obtain the target category and target coordinates, including:

[0046] The fused feature map is input into the recognition head of the fused convolutional recognition network. The recognition head is a dual-branch detection and classification structure, which includes a grid classification branch and a grid regression branch.

[0047] The grid classification branch is used to perform grid local feature aggregation on the fused feature map, and the target class probability of each grid is output through convolution mapping to obtain the multi-target class distribution;

[0048] Multiple sets of anchor frames are preset. Based on these preset sets of anchor frames, the four-dimensional coordinate offset of the effective target region in the fused feature map is calculated grid by grid using the grid regression branch. Combined with the grid position and the anchor frame reference, the independent coordinate information of multiple targets is decoded. Finally, the target category and target coordinate set corresponding to all effective targets in the fused feature map are output.

[0049] In a second aspect, the present invention provides an edge image recognition chip acceleration system based on multi-scale fusion, wherein the acceleration system is applied to the acceleration method of the first aspect, and the acceleration system includes:

[0050] The image acquisition and feature extraction module is used to acquire raw image data of the target area using an image sensor, normalize and scale the raw image data to obtain standard image data, construct a fusion convolutional recognition network, and input the standard image data into the fusion convolutional recognition network for forward inference to obtain multi-scale raw features, which include a first feature map, a second feature map and a third feature map.

[0051] The feature block entropy filtering and recombination module is connected to the image acquisition and feature extraction module. The feature block entropy filtering and recombination module is used to preset size threshold and entropy threshold, divide the first feature map and the second feature map into multiple non-overlapping feature blocks according to the size threshold, calculate the scale information entropy of each non-overlapping feature block, compare multiple scale information entropy with entropy threshold to obtain effective target blocks, smooth background blocks and blank invalid blocks, sample and compress the smooth background blocks to obtain compressed background blocks, and discard blank invalid blocks, recombine the effective target blocks and compressed background blocks in the first feature map to obtain the filtered first feature, and recombine the effective target blocks and compressed background blocks in the second feature map to obtain the filtered second feature.

[0052] The hierarchical cache parallel scheduling module is connected to the feature block entropy filtering and reorganization module. The hierarchical cache parallel scheduling module is used to write the effective target blocks in the first and second filtered features into the feature micro-cache using a double-buffered ping-pong read-write mechanism, write the compressed background blocks in the first and second filtered features into a temporary FIFO, and store the third feature map into the semantic cache. The double-buffered ping-pong read-write mechanism can perform data writing and data reading operations in parallel.

[0053] The dynamic fusion weight module is connected to the hierarchical cache parallel scheduling module. The dynamic fusion weight module is used to construct a bidirectional cross-scale interactive branch in the fusion convolutional recognition network. It reads the filtered first feature, filtered second feature and third feature map from the feature micro-cache, temporary FIFO and third feature map respectively. It uses the bidirectional cross-scale interactive branch to solve the scale cross-correlation matrix of the filtered first feature, filtered second feature and third feature map and generates dynamic fusion weights in real time.

[0054] The fusion feature map construction module is connected to the dynamic fusion weight module. The fusion feature map construction module is used to perform weighted fusion of the first feature, the second feature and the third feature map after screening based on the dynamic fusion weight to obtain the fusion feature map. When calculating the scale information entropy, the scale cross-correlation matrix and the fusion feature map, a multi-operator parallel hybrid computing method is used for synchronous operation, and computing resources are dynamically allocated according to the number of effective target blocks.

[0055] The recognition head inference output module is connected to the fusion feature map construction module. The recognition head inference output module is used to input the fusion feature map into the recognition head of the fusion convolutional recognition network, and use the recognition head to classify and detect the target in the fusion feature map to obtain the target category and target coordinates.

[0056] This application provides a method and system for accelerating edge image recognition chips based on multi-scale fusion. The method acquires original images and preprocesses them to obtain standard image data, which is then input into a fusion convolutional recognition network to extract three layers of multi-scale original features. The scale information entropy is calculated for shallow and medium-layer feature maps in blocks to distinguish effective target blocks, smooth background blocks, and blank invalid blocks. After compressing the background blocks and removing invalid blocks, the filtered features are obtained by recombining them. A double-buffered ping-pong read-write mechanism is used to cache different feature partitions, and a bidirectional cross-scale interactive branch is constructed to read the cached features. The scale cross-correlation matrix is ​​solved to generate dynamic fusion weights and obtain fusion feature maps. The operation process adopts multi-operator parallel computation and dynamically schedules computing power according to the number of effective target blocks. Finally, the recognition head completes target classification and coordinate detection, reducing the redundant overhead of edge chip feature data and balancing recognition accuracy and inference speed. Attached Figure Description

[0057] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0058] Figure 1 A flowchart illustrating an edge image recognition chip acceleration method based on multi-scale fusion provided in this application;

[0059] Figure 2 This application provides a connection diagram for an edge image recognition chip acceleration system based on multi-scale fusion. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without creative effort falls within the scope of protection of this application.

[0061] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0062] In this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0063] This application provides an acceleration method and system for edge image recognition chips based on multi-scale fusion. The method preprocesses the acquired raw image and then feeds it into a fusion convolutional recognition network to extract three layers of multi-scale raw features. For shallow and medium-level features, it performs block segmentation and scale information entropy determination, filters and retains effective target blocks, compresses and smooths background blocks, and removes blank and invalid blocks. Differentiated partition caching is achieved through a double-buffered ping-pong read-write mechanism. Features are read by relying on bidirectional cross-scale interactive branches, and the scale cross-correlation matrix is ​​solved to generate dynamic fusion weights to complete feature fusion. The entire operation adopts a multi-operator parallel mode and dynamically allocates computing power according to the number of effective target blocks. Finally, the recognition head realizes target classification and location detection, effectively reducing feature redundancy and improving the inference performance of edge chip image recognition.

[0064] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0065] Figure 1 A flowchart illustrating an edge image recognition chip acceleration method based on multi-scale fusion provided in this application is shown below. Figure 1As shown in this embodiment, an edge image recognition chip acceleration method based on multi-scale fusion is provided. This acceleration method includes: acquiring raw image data of the target region using an image sensor; normalizing and scaling the raw image data to obtain standard image data; constructing a fusion convolutional recognition network; inputting the standard image data into the fusion convolutional recognition network for forward inference to obtain multi-scale raw features, wherein the multi-scale raw features include a first feature map, a second feature map, and a third feature map; setting a size threshold and an entropy threshold; dividing the first feature map and the second feature map into multiple non-overlapping feature blocks according to the size threshold; and calculating... The scale information entropy of each non-overlapping feature block is calculated. Multiple scale information entropies are compared with an entropy threshold to obtain valid target blocks, smooth background blocks, and blank invalid blocks. Smooth background blocks are sampled and compressed to obtain compressed background blocks, and blank invalid blocks are discarded. Valid target blocks and compressed background blocks in the first feature map are recombined to obtain the filtered first feature. Valid target blocks and compressed background blocks in the second feature map are recombined to obtain the filtered second feature. A double-buffered ping-pong read / write mechanism is used to write the valid target blocks from the filtered first and second features into a feature micro-buffer, and the compressed background blocks from the filtered first and second features into a temporary buffer. FIFO (First-In, First-Out) stores the third feature map in a semantic buffer, where a double-buffered ping-pong read / write mechanism allows for parallel data writing and reading operations. A bidirectional cross-scale interaction branch is constructed in the fused convolutional recognition network. Valid target blocks of the first and second selected features are retrieved from the feature micro-buffer, and compressed background blocks of the first and second selected features are retrieved from the temporary FIFO. These are then concatenated and recombined according to their spatial positions to obtain complete first and second selected features. Finally, the third feature map is read from the semantic buffer and processed using the bidirectional cross-scale interaction... The cross-correlation matrix of the first, second, and third feature maps after screening is solved by inter-branch calculation, and dynamic fusion weights are generated in real time. Based on the dynamic fusion weights, the first, second, and third feature maps after screening are weighted and fused to obtain a fused feature map. When calculating the scale information entropy, scale cross-correlation matrix, and fused feature map, a multi-operator parallel hybrid computing method is used for synchronous operation, and computing resources are dynamically allocated according to the number of effective target blocks. The fused feature map is input into the recognition head of the fused convolutional recognition network, and the recognition head is used to classify and detect targets in the fused feature map to obtain the target category and target coordinates.

[0066] This application provides a method and system for accelerating edge image recognition chips based on multi-scale fusion. The method acquires raw image data of the target region and obtains standard image data through normalization and scaling. This standard image data is then input into a fusion convolutional recognition network to infer multi-scale raw features composed of a first feature map, a second feature map, and a third feature map. The first and second feature maps are divided into blocks, and scale information entropy is calculated to distinguish effective target blocks, smooth background blocks, and blank invalid blocks. After compressing the smooth background blocks and discarding invalid blocks, the filtered first and second features are recombined. A double-buffered ping-pong read-write mechanism is used to store different types of features into a feature micro-buffer, a temporary FIFO, and a semantic buffer, respectively, achieving parallel read and write operations. Various cached features are read through bidirectional cross-scale interactive branches within the fusion convolutional recognition network. The scale cross-correlation matrix is ​​solved to generate dynamic fusion weights in real time, completing the multi-scale feature weighted fusion to obtain a fused feature map. The computation process employs multi-operator parallel computation and dynamically allocates computational resources based on the number of effective target blocks. Finally, the fused feature map is fed into a recognition head for classification and target detection, outputting the target category and target coordinates.

[0067] This application constructs a fused convolutional recognition network, which is divided into four main functional modules: an image input preprocessing layer, a multi-scale feature extraction backbone network, a bidirectional cross-scale dynamic fusion branch, and a dual-branch recognition output head. The network adopts a lightweight, progressive structure, with a standard 416×416×3 RGB image as input and target category probability and target pixel coordinates as output. The overall architecture avoids redundant deep stacking, balancing feature extraction capability with edge-end inference speed, and adapts to the acceleration mechanisms of feature entropy filtering, hierarchical caching scheduling, and dynamic computing power allocation in this invention. Specifically, the backbone network is responsible for outputting three layers of differentiated multi-scale original features; the cross-scale fusion branch is responsible for feature association modeling and dynamic weight generation; and the recognition head is responsible for final detection, classification, and coordinate regression, forming a complete end-to-end architecture of "input-multi-scale extraction-filtering caching-dynamic fusion-inference output".

[0068] In the input layer of this network, the input size is uniformly 416×416×3. It receives standard image data that has been normalized and scaled by bilinear interpolation. Its purpose is to provide the network with inputs of uniform specifications and stable pixel distribution, eliminate data differences caused by acquisition hardware and ambient lighting, and ensure the stability of subsequent convolutional feature extraction.

[0069] The multi-scale feature extraction backbone network includes shallow ordinary convolutional feature extraction layers, mid-level depth-separable convolutional feature extraction layers, and deep dilated convolutional feature extraction layers. The shallow ordinary convolutional feature extraction layers employ 3×3 standard ordinary convolutions with 64 kernels, a stride of 1, and padding of 1, combined with batch normalization (BN) layers and the ReLU activation function. No downsampling is performed, and the output feature map size is 416×416×64, which is the first feature map. Through dense ordinary convolutions, high-precision extraction of shallow textures, edges, and contour details of the image is achieved, preserving fine-grained feature information of small targets. The high feature resolution and rich detail provide a high-precision feature basis for subsequent feature block segmentation, scale entropy calculation, and effective target block selection. The mid-layer depthwise separable convolutional feature extraction layer uses 3×3 depthwise separable convolutions with 64 depthwise convolution kernels, 128 pointwise convolution kernels, a stride of 2, and padding of 1. Combined with BN normalization and ReLU activation, it performs one downsampling, outputting a feature map of size 208×208×128, which is the second feature map. Building upon shallow detail features, a lightweight convolutional structure is used to perform feature upscaling and purification. While significantly reducing the number of computational parameters and computational load, it retains the mid-layer target contour and local semantic information, serving as a transitional feature between shallow details and deep semantics, balancing network accuracy and inference speed, and adapting to the low-power computing requirements of edge computing. The deep dilated convolutional feature extraction layer uses 3×3 dilated convolutions with a dilation rate r=2, 256 kernels, a stride of 2, and padding of 2. Combined with BN normalization and ReLU activation, it performs a second downsampling, outputting a feature map of size 104×104×256, which is the third feature map. By expanding the receptive field to capture global contextual information and long-distance pixel dependencies in images, we focus on extracting overall semantic features of large targets and global features of the scene, providing deep semantic support for cross-scale semantic transmission and global feature fusion.

[0070] The bidirectional cross-scale dynamic fusion branch includes a top-down semantic propagation branch and a bottom-up detail feedback branch. It does not involve additional convolutional downsampling and only includes feature concatenation, matrix correlation calculation, and weight normalization modules. The top-down branch propagates global semantic information from the deep third feature map to the shallow layer, compensating for insufficient semantic information in the shallow features. The bottom-up branch feeds back shallow and mid-level detailed features to the deep layer, compensating for missing details in the deep features. Multi-scale feature correlation modeling is achieved through scale cross-correlation matrix calculation, adaptively generating dynamic fusion weights to complete the intelligent weighted fusion of the three layers of features.

[0071] The dual-branch recognition output head consists of a grid classification branch and a grid regression branch. Both branches employ a 1×1 convolutional dimensionality reduction + fully connected layer mapping structure, with no redundant convolution stacking. Global feature aggregation is performed on the fused feature map to complete target category probability prediction. The grid regression branch calculates the effective region coordinate offset of the target, accurately determining the target center coordinates and boundary positions.

[0072] The construction process of the overall fusion convolutional recognition network is as follows: First, a network input preprocessing interface is built, fixing the network input size to 416×416 to match the front-end image preprocessing output specifications, completing input data dimensionality verification and normalization adaptation configuration. Second, shallow ordinary convolutional modules, mid-layer depth-separable convolutional modules, and deep dilated convolutional modules are stacked sequentially to construct a three-level multi-scale feature extraction backbone, outputting high-resolution detail features, lightweight transition features, and global semantic features, forming a complete multi-scale feature system. Third, a bidirectional cross-scale interaction branch is embedded to establish a bidirectional information transmission path for the three-layer features, configuring scale cross-correlation matrix operation units and dynamic weight generation units to achieve a feature adaptive fusion mechanism. Fourth, a hierarchical caching adaptation interface is configured to match the feature output end with a feature micro-caching, temporary FIFO, and semantic cache partitioned storage mechanism, adapting to the ping-pong read / write parallel scheduling logic. Fifth, a dual-branch recognition head is built, configuring classification and regression inference branches respectively, completing the detection and classification functions of the network output end, ultimately forming a complete end-to-end fusion convolutional recognition network.

[0073] The fusion convolutional recognition network employs a training method combining transfer learning and end-to-end fine-tuning, balancing training efficiency and recognition accuracy, and is suitable for edge detection scenarios. The specific training process is as follows:

[0074] Dataset construction and preprocessing: The network is trained using a mixture of publicly available object detection datasets and self-built scene datasets, which include object samples of various sizes. The samples are randomly cropped, flipped, and color gamut transformed to augment the data and improve the network's generalization ability. All sample sizes are standardized to be consistent with the input specifications of this network.

[0075] Pre-trained weight initialization: Parameter initialization is performed based on the pre-trained weights of the lightweight convolutional network, which accelerates network convergence, avoids training oscillations caused by random initialization, and preserves the basic image feature extraction capability.

[0076] Loss function configuration: Training uses a joint loss function, including classification cross-entropy loss and coordinate regression smoothing L1 loss, jointly constraining network training. The formula is: .in The target classification loss is used to constrain the accuracy of category prediction; This is the coordinate regression loss, used to constrain the target positioning accuracy.

[0077] Hyperparameter configuration: training batch size is set to 16, initial learning rate is set to 1e-3, cosine annealing learning rate decay strategy is adopted, total number of iterations is 100, and weight decay coefficient is set to 5e-4 to prevent network overfitting.

[0078] Layered training and fine-tuning: First, freeze the weights of the backbone network and train the bidirectional cross-scale fusion branch and the recognition head separately to adapt to the dynamic weight fusion mechanism; then unfreeze all network parameters and fine-tune the weights of all layers globally to make multi-scale feature extraction, feature interaction fusion, and detection inference form the optimal matching state.

[0079] Model convergence and quantization derivation: Training stops when the accuracy of the validation set no longer improves and the loss function tends to stabilize, thus completing model convergence; edge quantization compression is performed on the trained network to adapt to the hardware inference architecture of the image recognition chip, ensuring high-speed and low-latency inference after deployment.

[0080] This method effectively reduces data redundancy in feature transmission and computation by filtering out invalid features and compressing background features through scale information entropy; it improves data throughput efficiency by using hierarchical partitioning caching combined with a ping-pong read-write mechanism; it adapts to different target distributions by relying on bidirectional cross-scale interaction and dynamic fusion weights; and it fully utilizes the computing power of edge chips by combining multi-operator parallel operation and dynamic computing power scheduling. This can reduce inference latency and hardware power consumption while ensuring target recognition accuracy, thus achieving efficient acceleration of image recognition tasks at the edge.

[0081] Specifically, the original image data of the target area is acquired using an image sensor, and the original image data is normalized and scaled to obtain standard image data. This includes: setting a preset resolution and frame rate, acquiring the original image data of the target area using an image sensor based on the resolution and frame rate; mapping the pixel values ​​of the original image data uniformly to the standard range of [0,1] to obtain normalized image data; setting a preset standard size, scaling the normalized image data to the standard size using a bilinear interpolation algorithm while preserving edge details and texture features to obtain standard image data.

[0082] This application provides an edge image recognition chip acceleration method and system based on multi-scale fusion. The method collects the original image data of the target area according to a preset resolution and frame rate, normalizes the collected image pixels, maps the pixel values ​​to the [0,1] interval, and uses a bilinear interpolation algorithm to uniformly scale the image to the network standard input size, while fully preserving the image edge details and texture information, and finally obtains standard image data with uniform size and regular pixel distribution, providing standardized input for subsequent feature extraction.

[0083] The specific steps are as follows: The image sensor's acquisition resolution and output frame rate are preset. In this embodiment, the acquisition resolution is set to 640×480 pixels, and the output frame rate is 30fps, meaning 30 consecutive images are acquired per second. The image sensor continuously samples the target monitoring area in real time according to the preset resolution and frame rate, outputting 8-bit RGB three-channel raw image data. The value range of a single-channel raw pixel is a fixed integer interval. .in, The pixel coordinates of the original image in a two-dimensional plane. The pixel number is in the horizontal direction. The vertical pixel number.

[0084] During the acquisition process, the image sensor transmits the acquired raw image data stream frame by frame to the on-chip buffer of the edge chip, completing the hardware acquisition and data caching of the raw image data and providing raw data support for subsequent software preprocessing. Affected by ambient light intensity, automatic lens exposure, and sensor gain fluctuations, the raw images acquired at different times have differences in brightness and contrast, and the pixel distribution is highly discrete, making them unsuitable for direct input into convolutional network inference and requiring standardized preprocessing.

[0085] To eliminate differences in pixel amplitude, illumination interference, and exposure deviations in the original image, unify the pixel data distribution range, and improve the stability and convergence of subsequent network inference, a global pixel normalization operation is performed on the acquired original image data. This maps the discrete integer pixel values ​​from 0 to 255 to the floating-point standard range of [0,1]. The normalization calculation formula is as follows:

[0086] ;

[0087] in, This represents the original pixel grayscale value at coordinates (x, y) in the original image, with a value range of [0, 255], and is an 8-bit unsigned integer; This indicates the image after normalization in coordinates. The standardized pixel value at the location is strictly constrained to the range [0,1] and is floating-point data; This represents the maximum extreme value of the pixel values ​​in an 8-bit image, and is a normalized fixed scaling factor.

[0088] Through the above normalization operation, the pixel amplitude deviation caused by hardware acquisition and environmental interference in different frames of images is eliminated, the pixel data distribution pattern of all input images is unified, and normalized image data with pixel standardization is obtained, which effectively improves the robustness of forward inference of the fusion convolutional recognition network.

[0089] Convolutional recognition networks have a fixed model input size, and the original resolution images acquired by image sensors cannot be directly adapted to the network input specifications. Therefore, the normalized images need to be scaled. This invention uses a bilinear interpolation algorithm to perform image scaling. Compared with nearest neighbor interpolation, it can effectively preserve image edge contours and texture details, avoiding scaling distortion and loss of details, and balancing size adaptability with image feature integrity.

[0090] First, a standard input size for the network is preset. For example, in this embodiment, the standard size is set to 416×416 pixels, and normalized images of any resolution are uniformly scaled to this standard size. Bilinear interpolation reconstructs the pixel value by weighting the four adjacent original pixels around the target pixel, and simultaneously achieves smooth interpolation by combining horizontal and vertical bidirectional offset coefficients. The core calculation formula is as follows:

[0091] ;

[0092] in, This represents the reconstructed pixel value at the target coordinates (x, y) in the scaled standard image; This represents the top-left, top-right, bottom-left, and bottom-right reference pixel values ​​adjacent to the target point to be interpolated in the normalized original image; This represents the horizontal offset coefficient of the target point to be interpolated relative to the four reference pixels, with a value range of [0,1], representing the proportion of horizontal pixel offset. This represents the vertical offset coefficient of the target point to be interpolated relative to the four reference pixels, with a value range of [0,1], representing the proportion of pixel offset in the vertical direction. , Calculated by mapping the pixel coordinates of the images before and after scaling:

[0093] Let the size of the normalized image (interpolation source image) be... The target pixel coordinates of the scaled standard image are The floating-point coordinates of the target pixel mapped back to the source image are: ,

[0094]

[0095] in, To scale the output standard image's width and height; take Round down to obtain the horizontal reference integer coordinates of the source image. , Rounding down yields the vertical reference integer coordinates of the source image. The horizontal offset coefficient and the vertical offset coefficient are then:

[0096]

[0097] in, That is, interpolating four reference pixels. The corresponding integer coordinates of the top-left pixel in the source image; thus, by using the target output pixel coordinates, the solution can be uniquely obtained. and This ensures that the value strictly falls within the interval [0,1].

[0098] Through the complete preprocessing workflow of hardware parameter configuration, raw image acquisition, pixel normalization, and bilinear interpolation scaling, standard image data with uniform size, standardized pixel distribution, and complete detailed features is finally obtained. This standard image data is fully compatible with the input specifications of the fused convolutional recognition network, completely solving the problems of inconsistent original image resolution, discrete pixel distribution, and unstable input data caused by environmental interference. It provides high-precision and high-stability raw input support for subsequent core steps such as multi-scale feature layer extraction, feature entropy filtering, and cross-scale feature fusion.

[0099] This method can eliminate interference from differences in lighting, exposure, and resolution of the original image, and improve the stability of network inference by unifying the image input specifications. At the same time, the bilinear interpolation scaling method can preserve the image details to the maximum extent while adapting the size, avoiding information loss, and providing a high-quality and consistent image input foundation for subsequent multi-scale feature extraction, target screening, and fusion recognition.

[0100] Specifically, standard image data is input into a fusion convolutional recognition network for forward inference to obtain multi-scale original features, including: firstly, using ordinary convolutional layers in the fusion convolutional recognition network to extract the first feature map from the standard image data; secondly, using separable convolutional layers in the fusion convolutional recognition network to extract the second feature map from the standard image data; and finally, using dilated convolutional layers in the fusion convolutional recognition network to extract the third feature map from the standard image data.

[0101] This application provides an edge image recognition chip acceleration method and system based on multi-scale fusion. The method feeds standard image data into a fusion convolutional recognition network to perform forward inference. It uses ordinary convolutional layers in the network to extract a first feature map containing image detail information, uses depth-separable convolutional layers to extract transition features to form a second feature map, and expands the receptive field through dilated convolutional layers to extract a third feature map with global semantic information, thereby obtaining three layers of multi-scale original features.

[0102] The fusion convolutional recognition network adopts a hierarchical, progressive, multi-scale feature extraction architecture. For standard image data, different functional convolutional modules are sequentially used to complete feature mapping. Specifically, ordinary convolutions accurately extract shallow texture and edge details, generating high-resolution detail feature maps; depthwise separable convolutions perform mid-level feature transformations with lightweight computation, preserving effective feature information while reducing computational load; and dilated convolutions expand the receptive field, capturing global semantics and long-range dependencies in the image to generate deep global feature maps. These three types of convolutions work collaboratively to achieve full coverage of multi-scale features, including fine-grained details, mid-grained transitions, and coarse-grained semantics.

[0103] The first feature map is the shallow output feature of the network, which mainly preserves fine-grained details such as image edges, textures, and contours. It is suitable for the feature extraction needs of small targets and is extracted by a standard two-dimensional ordinary convolutional layer. The core formula of the convolution operation is as follows:

[0104] ;

[0105] in, This indicates the output coordinates in the first feature map. The characteristic response value of the location; This represents the standard image data input. This refers to the image channel dimension, typically RGB three channels; Indicates the total number of input feature channels; This indicates the size of a standard convolution kernel; in this embodiment, it is preferred. Standard convolution kernel; This represents the weight parameters of a regular convolution kernel. The coordinates are relative to each other within the convolution kernel. This represents the bias term of a regular convolutional layer, used to fine-tune the baseline value of the feature response.

[0106] Ordinary convolution completes dense sampling of local image features by sliding the convolution kernel pixel by pixel. It can preserve shallow details of the image to the greatest extent, without losing any feature details. Finally, it outputs a first feature map with high resolution and rich details, providing a core basis for subsequent selection of effective feature blocks for small targets.

[0107] The second feature map is a transitional feature in the middle layer of the network, connecting shallow detailed features and deep semantic features. It achieves feature dimensionality compression and feature information purification, balancing feature richness and computational cost. Extraction is completed by a depthwise separable convolutional layer. The depthwise separable convolution is divided into two steps: depthwise convolution and pointwise convolution, significantly reducing the number of parameters and computational cost, adapting to the lightweight inference needs of edge chips. The formula for depthwise convolution (single-channel spatial feature extraction) is as follows:

[0108] ;

[0109] in, This represents the intermediate feature map output by the depthwise convolution; This embodiment preferably represents the depth-separable convolution kernel size. ; This represents the weights of the depthwise convolution kernel, which performs convolution only on single-channel spatial features and does not perform channel fusion. The formula for pointwise convolution (channel feature fusion) is as follows:

[0110] ;

[0111] in, This represents the coordinates of the second feature map in the final output. Location feature value; This represents a 1×1 pointwise convolution weight, used to achieve feature fusion and dimension transformation in the channel dimension; This represents the bias term for depth-separable convolutional layers.

[0112] This convolution method first extracts single-channel spatial features and then fuses channel features, which can reduce the amount of computation by more than 80% compared with ordinary convolution. At the same time, it fully preserves the intermediate transition features and generates a second feature map that balances accuracy and efficiency, making it suitable for low-power inference scenarios at the edge.

[0113] The third feature map represents the deep semantic features of the network, used to capture global contextual information of the image, overall features of large targets, and long-distance pixel dependencies. It addresses the limited receptive field of ordinary convolutions and is extracted by dilated convolutional layers. Dilated convolution increases the receptive field by setting the dilation rate, eliminating the need for pooling downsampling and avoiding feature resolution loss. The core calculation formula is as follows:

[0114] ;

[0115] in, This indicates the coordinates in the third feature map. Global semantic feature values ​​of location; This indicates the basic size of the dilated convolution kernel; in this embodiment, it is preferred. ; This represents the dilation rate of the convolution (hole rate), which is set to 2 in this embodiment to expand the convolution sampling receptive field; This represents the weight parameters of the dilated convolution kernel; This represents the bias term of the dilated convolution layer.

[0116] Dilated convolution expands the receptive field by sampling pixels at intervals. Without reducing feature resolution or losing feature information, it effectively captures global semantic information of the image and outputs a third feature map with strong contextual relevance, providing support for subsequent cross-scale semantic interaction and large target recognition.

[0117] Through the aforementioned three layers of differentiated convolution operations, three complementary multi-scale original features are ultimately output: the first feature map carries shallow, fine-grained detail information, the second feature map carries mid-level, lightweight transitional feature information, and the third feature map carries deep, global semantic information. These three types of features form a complete multi-scale feature system, from local details to global semantics, and from high-resolution lightweight features to strong semantic features. This effectively compensates for the information deficiencies of single-scale feature extraction and provides comprehensive, multi-level feature inputs for subsequent feature block selection, bidirectional cross-scale feature fusion, and adaptive weight optimization.

[0118] This method extracts multi-scale features by dividing the work among different types of convolutional layers. Ordinary convolution fully preserves shallow texture details, depthwise separable convolution completes feature transformation while controlling computational load, and dilated convolution expands the receptive field by dilating the sampling interval. The dilation kernel itself does not actively compress feature resolution and does not rely on continuous pooling to expand the receptive field. In this embodiment, the dilated convolution stride is manually set to stride=2. The feature map size is downsampled by a controllable stride, compressing the 208×208 second feature map into a 104×104 third feature map. This constructs a three-level multi-scale feature with differentiated resolution. The resolution reduction is only controlled by the convolution stride and is not caused by the dilation characteristics. The three types of features work together to take into account both fine-grained target information and global context, providing a rich feature foundation for subsequent feature selection and cross-scale fusion.

[0119] Specifically, the first feature map and the second feature map are divided into multiple non-overlapping feature blocks according to a size threshold, and the scale information entropy of each non-overlapping feature block is calculated, including: setting a preset size threshold, dividing the first feature map and the second feature map into multiple non-overlapping feature blocks based on the feature map resolution and the size threshold; traversing multiple non-overlapping feature blocks, and calculating the scale information entropy of each non-overlapping feature block separately using a channel-by-channel independent calculation strategy.

[0120] This application provides an edge image recognition chip acceleration method and system based on multi-scale fusion. The method pre-sets a size threshold, combines the resolution of the first feature map and the second feature map to divide the two feature maps into several non-overlapping feature blocks, traverses all feature blocks and calculates the scale information entropy corresponding to each non-overlapping feature block.

[0121] To achieve refined information statistics in local areas and avoid the problem that global feature statistics cannot distinguish between local targets and background, this invention performs non-overlapping and full-coverage rule-based block processing on the first feature map and the second feature map. The specific implementation method is as follows.

[0122] Preset fixed size threshold , This represents the side length of a single feature block, in pixels. In this embodiment, it is preferred. That is, all feature blocks are unified as A square region. Based on the resolution of the first feature map. Second feature map resolution The two feature maps are divided equally in both the horizontal and vertical directions to obtain several local non-overlapping feature blocks of the same size that do not overlap and are fully covered.

[0123] Taking a single feature map as an example, the feature map height is Width is Number of horizontal blocks after partitioning With the number of vertical blocks The calculation formula is:

[0124] ;

[0125] ;

[0126] in, These represent the height and width in pixels of the current feature map to be segmented; This represents the preset block size threshold, which characterizes the side length of a single non-overlapping feature block; Indicates the total number of horizontal blocks; Indicates the total number of vertical blocks; This represents the floor function, ensuring that the block size is strictly uniform and there are no irregular edge blocks.

[0127] After the segmentation is completed, the entire feature map is divided into segments. Each feature block is a standard square feature block, and each feature block independently participates in subsequent grayscale statistics, texture gradient statistics and information entropy calculation, so as to realize the local fine decomposition of global features.

[0128] For each independent non-overlapping feature block, the distribution pattern of all pixel feature response values ​​within the block is statistically analyzed, and the gray-level distribution probability is calculated to characterize the dispersion and information richness of pixel amplitudes within the feature block.

[0129] Let any feature block region be The block contains a total of 100 pixels. The feature pixel response value is Statistical eigenvalues Frequency of occurrence within this feature block The formula for calculating the probability of the corresponding gray-scale distribution is:

[0130] ;

[0131] in, Represents eigenvalues The corresponding grayscale distribution probability; This indicates that the feature value within the current feature block is The total number of pixels; Represents the total number of pixels within a single feature block, which is fixed at 1. .

[0132] If the feature block is a blank invalid background area, the pixel feature values ​​tend to be consistent and the probability distribution is simple; if the feature block contains target texture and edges, the pixel feature values ​​are discrete, the probability distribution is more uniform, and the information content is higher.

[0133] This method iterates through multiple non-overlapping feature blocks and calculates the scale information entropy of the multi-channel floating-point feature tensors, abandoning the traditional image grayscale statistical method and adapting to the positive and negative floating-point responses and continuous value distribution characteristics of convolutional features. For each feature block, a channel-by-channel independent calculation strategy is adopted, without pre-aggregating across channels. First, intra-block Min-Max normalization is performed on all floating-point feature values ​​within a single channel, uniformly mapping feature responses containing negative values ​​to the positive value interval [0,1]. A statistical histogram is constructed for the normalized continuous floating-point features using a fixed uniform binning method. The frequency and probability distribution of feature response distribution are obtained by interval counting, solving the problem of continuous floating-point features having no repeated values ​​and being unable to statistically calculate probabilities. Texture gradient distribution features are calculated within the single-channel spatial dimension to independently represent the local texture richness of each channel. After calculating the unit information entropy channel by channel, the mean of all channel entropy values ​​is calculated to obtain the original scale information entropy of the current feature block.

[0134] The formula for correcting and normalizing the negative value of floating-point features is as follows:

[0135] ;

[0136] in, For the first The original floating-point characteristic value of the channel (can be positive or negative); This represents the maximum or minimum value of a single channel feature in the current block. This is the normalized positive value feature after correction.

[0137] The binning probability statistics for continuous floating-point feature histograms are as follows:

[0138] ;

[0139] Among them, [0,1] is uniformly divided into fixed Each interval For the first Number of feature pixels within the interval This represents the total number of pixels within the block. This represents the probability of the corresponding interval.

[0140] The original Shannon entropy for a single channel is:

[0141] ;

[0142] in, To prevent the minimum value Meaningless.

[0143] Key correction: Normalized scaled information entropy (forced range [0,1]) is:

[0144] ;

[0145] ;

[0146] in, The maximum theoretical entropy, the normalized single-channel entropy ; The block-level normalized scale information entropy used for the final threshold determination has a strictly fixed range of [0,1].

[0147] This method achieves regular block division of feature maps by fixed size, which facilitates independent determination of local region features in the future; scale information entropy can objectively distinguish between target region and background region, providing a reliable quantitative basis for subsequent feature block classification, background compression and invalid block removal.

[0148] Specifically, multiple scale information entropies are compared with entropy thresholds to obtain valid target blocks, smooth background blocks, and blank invalid blocks. Smooth background blocks are sampled and compressed to obtain compressed background blocks, and blank invalid blocks are discarded. Valid target blocks and compressed background blocks in the first feature map are recombined to obtain the filtered first feature. Valid target blocks and compressed background blocks in the second feature map are recombined to obtain the filtered second feature. This includes: setting a first entropy threshold and a second entropy threshold; determining non-overlapping feature blocks with scale information entropy greater than or equal to the first entropy threshold as valid target blocks; and determining non-overlapping feature blocks with scale information entropy greater than or equal to the second entropy threshold and less than the first entropy threshold as valid target blocks. The smooth background block is identified as a smooth background block; non-overlapping feature blocks with scale information entropy less than the second entropy threshold are identified as blank invalid blocks; the smooth background block is sampled and compressed using the mean downsampling compression algorithm to obtain a compressed background block, while discarding blank invalid blocks; according to the spatial positional relationship between the first feature map and the second feature map, the valid target block in the first feature map is recombined with the compressed background block to obtain the filtered first feature, and the valid target block in the second feature map is recombined with the compressed background block to obtain the filtered second feature, wherein the filtered first feature and the filtered second feature are consistent with the feature size and spatial structure of the first feature map and the second feature map, respectively.

[0149] This application provides an edge image recognition chip acceleration method and system based on multi-scale fusion. The method pre-sets a first entropy threshold and a second entropy threshold, distinguishes effective target blocks, smooth background blocks, and blank invalid blocks according to the scale information entropy of the feature blocks, compresses the smooth background blocks using a mean downsampling compression algorithm and directly discards blank invalid blocks, and then recombines the effective target blocks and compressed background blocks with reference to the spatial position relationship of the original feature map to obtain the first and second filtered features with the same size and spatial structure as the original feature map.

[0150] To achieve accurate differentiation of feature blocks with different information contents, this invention presets two levels of entropy thresholds, namely the first entropy threshold. Second entropy threshold And satisfy the threshold relationship The first entropy threshold is the effective target discrimination threshold, used to filter target regions rich in texture and semantic information; the second entropy threshold is the invalid region discrimination threshold, used to distinguish between low-information smooth backgrounds and informationless blank regions. In this embodiment, considering the feature distribution characteristics of edge image recognition scenes, the following threshold is preferably set: .

[0151] Scale information entropy for all non-overlapping feature blocks The three types of feature blocks are quantitatively distinguished by threshold comparison. The specific discrimination rules are as follows: Valid target block discrimination: Smooth background block discrimination: ; Determining if a block is empty or invalid: .

[0152] in, The scale information entropy corresponding to a single non-overlapping feature block is calculated using the aforementioned gray-scale-texture fusion entropy formula; This represents the first entropy threshold, a high threshold used to define target feature regions with high information gain. This represents the second entropy threshold, a low threshold used to distinguish between low-information background regions and blank regions with no information.

[0153] Among them, the effective target block contains a large number of target edges, textures and details, which are the core effective features and must be fully preserved; the smooth background block contains only a small number of shallow features and no effective target information, and there is a lot of data redundancy, which can be compressed; the blank invalid block has no effective feature information, does not contribute to recognition and reasoning, and is directly discarded.

[0154] For the smooth background blocks obtained by the determination, this invention uses a 2×2 mean downsampling compression algorithm to simplify features, reducing the data volume while preserving the overall semantics of the background. Simultaneously, a nearest neighbor repeated broadcast upsampling mechanism is used to restore the size, ensuring that the size of the recombined feature region is completely consistent with the original 16×16 block. This method fixes the compression ratio k=2, uniformly subdivides the 16×16 smooth background block into 4 non-overlapping 2×2 sub-neighborhoods, and calculates the mean of the features in each sub-neighborhood to obtain 8×8 compressed background feature blocks. The mean downsampling compression calculation formula is as follows:

[0155] ;

[0156] in, Indicates the coordinates of the compressed feature block Output feature value at; Represents the coordinates of the neighborhood within the original smooth background block. The corresponding original feature response value; This indicates the downsampling compression ratio, which is fixed at 2 in this embodiment; Represents the pixel coordinates of the compressed feature block; Represents the neighboring pixel coordinates of the original smooth background block; For a fixed compression ratio.

[0157] To address the issues of reduced size and inability to match the original feature space structure after compression, this invention performs a nearest-neighbor repeated broadcast upsampling backfilling operation on the 8×8 compressed background block. This accurately maps the compressed data back to the original 16×16 feature block region, achieving size restoration. Specifically, each pixel value of the 8×8 compressed feature is repeatedly broadcast and filled to the corresponding pixel position in the original 2×2 sampled sub-neighborhood, completing the size restoration from 8×8 to 16×16. The backfilling mapping formula is as follows:

[0158] ;

[0159] in, This indicates the background feature values ​​that match the original 16×16 size after backfilling and restoration. The original feature block has 16×16 global coordinates; To compress the 8×8 local coordinates of the features, a precise mapping between the two coordinate systems is established to resolve the issue of inconsistent index dimensions. This backfilling method has extremely low computational cost, strong hardware adaptability, and can completely preserve the smooth semantic features of the compressed background without introducing additional texture noise.

[0160] The mean downsampling compression algorithm is used to perform a 2x downsampling on the smooth background block of size 16×16. When outputting the compressed background load of size 8×8, the upsampling size restoration operation is not performed on the compressed load in advance. The blank invalid blocks are directly marked with empty indices and the original pixel data is discarded. Only the 8×8 compressed background load is stored in the caching stage to save on-chip storage overhead. The nearest neighbor broadcast upsampling is only performed in real time in the cross-scale feature splicing and recombination stage to restore the 8×8 compressed background load to a 16×16 standard background block. According to the spatial position relationship between the first feature map and the second feature map, the valid target block with the original size is completely preserved, the compressed background load to be restored in real time, and the blank invalid blocks marked with empty indices are logically recombined. The first feature and the second feature after filtering at the logical level are consistent with the feature size and spatial structure of the first feature map and the second feature map, respectively. The storage space corresponding to the 16×16 restored background block is not occupied in the physical storage medium.

[0161] By using mean downsampling compression, redundant and subtle feature data in the background region can be removed while preserving the overall distribution characteristics of the background, thus achieving lightweight background feature processing. At the same time, feature regions that are determined to be blank or invalid blocks are directly zeroed out and discarded, completely eliminating invalid feature data.

[0162] For blank and invalid blocks whose scale information entropy is less than the second entropy threshold, they are determined to be redundant regions without effective features or background semantics. Structure preservation and global zero-padding are performed; the feature block structure is not directly deleted, only the feature data is cleared, ensuring that the overall feature space dimension and topology remain unchanged. The final feature reorganization output formula is supplemented with a third type of region constraint, forming a complete three-stage judgment rule. The revised formula is as follows:

[0163] ;

[0164] in, The set of effective target block spatial regions fully preserves the original features; To smooth the set of background block spatial regions, the restored features after compression and backfilling are used for filling; This is a set of blank, invalid block spaces, uniformly filled with pixel values ​​of 0. Invalid block areas retain only a 16×16 standard block structure, do not participate in cached valid data storage, only occupy a fixed topological location, and consume no computing or storage resources.

[0165] To ensure that the filtered feature maps can fully adapt to the original network structure and subsequent fusion inference logic, this invention strictly follows the spatial position and topological relationship of the original first feature map and the second feature map to complete the precise splicing and recombination of the effective target block and the compressed background block, ensuring that the feature space structure remains unchanged and the dimensional size is uniform.

[0166] Feature reconstruction follows a one-to-one spatial correspondence principle: fully preserved valid target blocks and compressed background blocks are backfilled and stitched together according to the original block row and column arrangements, discarding all blank and invalid block regions. During reconstruction, a fill-in adaptation mechanism unifies feature dimensions, ensuring that the height, width, and number of channels of the reconstructed feature map are completely consistent with the original feature map. The reconstruction output formula is defined as follows:

[0167] ;

[0168] in, This represents the pixel feature values ​​of the final output feature map after filtering and recombination. This represents the feature values ​​of the valid target block that are not compressed and are fully preserved. This represents the feature value of the compressed background block after mean downsampling. This represents the set of spatial regions corresponding to the valid target blocks; This represents the set of spatial regions corresponding to the compressed background block.

[0169] To avoid spatial and temporal sequence disorder after feature selection and compression, and to ensure complete alignment between the FIFO buffer data stream and the original feature space arrangement, this invention employs a topological sequence anchoring mechanism to maintain the FIFO storage order. All feature blocks are assigned unique row and column topological numbers during the block partitioning stage, and their sequence positions are fixed according to a "row-first, column-later" order. During processing, smooth background blocks are compressed, backfilled, and restored before being written into a temporary FIFO according to their original topological numbers. Blank and invalid blocks are not written with any feature data, but their original sequence slots are retained; they are not deleted or have their sequence positions changed, only marked as empty. This mechanism completely preserves the spatial position and topological relationships of the original feature map, eliminating intra-frame feature misalignment and disorder while deleting invalid data, providing accurate temporal support for subsequent cross-scale position matching.

[0170] Before reading features and performing scale correlation calculations in the bidirectional cross-scale interactive branch, the system restores the global spatial position based on the preset topological numbering and coordinate mapping relationship of each feature block. The specific process is as follows: First, based on the row and column topological numbering of the feature blocks, the arrangement position of each 16×16 feature block in the overall feature map is restored; second, through the compression-backfill coordinate mapping formula, the local coordinates of all background blocks and target blocks are uniformly aligned to the global coordinate system of the original feature map; third, based on the ordered splicing of zero-filled invalid blocks, restored background blocks, and complete target blocks, a complete filtering feature is reconstructed that is completely consistent with the size, spatial structure, and positional distribution of the original feature map. Ultimately, this ensures that the spatial correspondence of all feature regions is accurately matched during the cross-scale cross-correlation matrix calculation and dynamic weight fusion process, without positional offset or regional disorder.

[0171] Special note: Compressed local coordinates are used in the feature compression stage. Calculate 8×8 compressed features to achieve data simplification; feature backfilling and recombination stages are performed through... The mapping relationship accurately converts local compressed coordinates into original global coordinates. All reorganization, splicing, and fusion operations uniformly adopt the original global coordinate system.

[0172] After classification, background compression, invalid block discarding, and spatial reorganization, the final filtered first and second features are obtained. The two filtered features strictly match the resolution, channel dimension, and spatial topology of the original first and second feature maps, without any dimensional transformation or spatial offset, and are fully compatible with subsequent hierarchical cache read / write and cross-scale feature fusion operations.

[0173] This step achieves a refined feature selection mechanism by differentiating feature regions with different information contents, which "preserves target features without loss, compresses background features with lightweight weight, and completely eliminates invalid features". Without losing effective information for target recognition, it reduces feature data redundancy to the greatest extent, reduces storage bandwidth and computing power consumption for edge image recognition chips, and ensures high-precision and low-latency inference results.

[0174] This method uses a dual-entropy threshold to achieve fine-grained differentiation of feature blocks, fully preserving information-rich target blocks, compressing redundant background blocks, and directly removing informationless blank blocks, significantly reducing the amount of data transmitted and processed in the future. The recombined features maintain the original spatial structure and do not destroy the correspondence in the feature space, thus reducing computing power and storage overhead while ensuring the effectiveness of subsequent multi-scale feature fusion.

[0175] Specifically, a double-buffered ping-pong read-write mechanism is used to write the effective target blocks from the first and second filtered features into a feature micro-cache, write the compressed background blocks from the first and second filtered features into a temporary FIFO, and store the third feature map into a semantic cache. This includes: storing the effective target blocks from the first and second filtered features into a small-capacity, fast-read / write feature micro-cache using a double-buffered ping-pong read-write mechanism; writing the compressed background blocks from the first and second filtered features into a temporary FIFO using a double-buffered ping-pong read-write mechanism; and storing the third feature map into a semantic cache using a double-buffered ping-pong read-write mechanism. The double-buffered ping-pong read-write mechanism includes a first buffer unit and a second buffer unit, which alternately perform data write and read operations. The feature micro-cache, temporary FIFO, and semantic cache are read and written independently without interference.

[0176] This application provides an edge image recognition chip acceleration method and system based on multi-scale fusion. The method adopts a double-buffered ping-pong read-write mechanism with a first buffer unit and a second buffer unit working alternately. The effective target blocks in the first and second features after screening are stored in a high-speed feature micro-buffer. The compressed background blocks in the first and second features after screening are stored in a temporary FIFO. The third feature map is stored in a semantic buffer. The feature micro-buffer, the temporary FIFO and the semantic buffer are read and written independently. The data writing and reading are executed in parallel by means of the buffer units working alternately.

[0177] The dual-buffered ping-pong read / write mechanism includes a first buffer unit and a second buffer unit, which alternately perform data writing and reading operations. The feature micro-buffer, temporary FIFO, and semantic buffer are three independent buffer partitions that are read and written independently without interference. The temporary FIFO uses a payload + meta-index paired storage structure. Each FIFO entry carries two types of data: an 8×8 compressed background payload and a global row and column topology index (block row number, block column number, and block type marker) of the feature block within the original feature map. The FIFO first-in-first-out mechanism only ensures the alignment of data read / write sequences within the same frame. During cross-scale feature reassembly, it does not rely on the FIFO read order to locate spatial positions but instead reads the built-in topology index of each data entry to complete accurate spatial mapping. Blank and invalid blocks do not write any payload data to the FIFO; they only retain empty placeholder indexes in the global topology record table to avoid spatial topology misalignment.

[0178] This invention constructs three types of functionally independent on-chip cache partitions with no interference between read and write operations. Each partition has differentiated hardware parameters, storage objects, and read / write characteristics to adapt to different data scales and real-time requirements. The specific partition configurations are as follows: Feature Micro-Cache: A small-capacity, ultra-high-speed on-chip cache dedicated to storing high-value, small-volume effective target block feature data, ensuring fast reading of target detail features and reducing latency for accessing key features; Temporary FIFO Cache: A first-in, first-out queue-style cache with continuous data stream throughput capability, dedicated to storing low-redundancy background feature blocks compressed by mean downsampling, adapting to pipelined read / write operations for batch background features; Semantic Cache: A medium-to-large capacity fixed cache partition dedicated to storing complete third-feature map global semantic features, used to stably preserve deep global features and ensure the integrity of cross-scale semantic interactions. The three types of cache partitions adopt an independent addressing and independent read / write bus architecture, ensuring that data does not overwrite or interfere with each other during operation, completely avoiding read / write conflicts between different types of feature data.

[0179] This invention introduces a double-buffered ping-pong read / write mechanism, setting up a first buffer unit with a consistent structure. With the second buffer unit The two buffer units alternately undertake data writing and data reading tasks, enabling parallel execution of read and write operations and eliminating the read and write waiting latency of the traditional single buffer mechanism.

[0180] Define the ping-pong scheduling state switching formula:

[0181] ;

[0182] in, This represents the first buffer unit, the basic storage unit of the double-buffered architecture; This indicates the second buffer unit, which pairs with the first buffer unit to complete the ping-pong alternating scheduling. This indicates the data write operation function, which completes the writing of feature data to the storage of the buffer unit; This represents the data read operation function, which completes the reading of characteristic data within the buffer unit. This indicates the timing count of the current inference frame, used to control the switching of the working state of the buffer unit; This represents the modulo operation, which is used to determine the parity of the timing state and ensure the alternation of the two units.

[0183] At any given time, this mechanism allows one buffer unit to receive new feature data being written, while another buffer unit is responsible for outputting historical data to subsequent fusion modules, thus achieving a parallel pipeline working mode of "writing new data and reading old data," which significantly improves the throughput efficiency of the feature data stream.

[0184] Based on a double-buffered ping-pong read / write mechanism, partitioned dedicated storage operations are performed on three different types of characteristic data to achieve accurate distribution and efficient caching of characteristic data.

[0185] Valid target blocks are written to the feature micro-cache: Valid target blocks from the first and second filtered features are crucial for identifying key details, requiring high data accuracy and real-time performance. Therefore, they are written separately to the high-speed feature micro-cache. Through alternating writing with dual buffer units, the target feature blocks extracted frame by frame are sequentially stored in the micro-cache address space, ensuring no loss of key features and allowing for fast random access by subsequent modules. Feature micro-cache storage mapping formula:

[0186] ;

[0187] in, This represents the valid target block feature data in the first feature after filtering; This represents the valid target block feature data in the second feature after filtering; This represents the dedicated storage address space for the feature micro-cache.

[0188] Compressed background blocks are written to a temporary FIFO buffer: The compressed background blocks in the first and second features after filtering are lightweight, redundant compressed data with small size and strong continuity, suitable for FIFO queue pipeline storage mode. They are written to the temporary FIFO buffer in batches using a double-buffered ping-pong mechanism, stored in an ordered manner according to the first-in-first-out rule, ensuring the temporal continuity of the background feature data stream and avoiding data out-of-order and stacking. Temporary FIFO buffer storage mapping formula:

[0189] ;

[0190] in, This represents the compressed background block data corresponding to the first feature after filtering; This represents the compressed background block data corresponding to the second feature after filtering; This represents the queue storage address space of the temporary FIFO buffer. A double-buffered ping-pong read / write mechanism is used to write the downsampled 8×8 compressed background payloads from the first and second filtered features into the temporary FIFO. The temporary FIFO only stores the compressed payloads and does not store the restored 16×16 background feature data.

[0191] The third feature map is written into the semantic cache: The unfiltered third feature map is a global semantic feature with a complete overall structure and fixed dimensions. It is used to provide cross-scale global semantic information. It is written entirely into an independent semantic cache for fixed storage of the entire map, ensuring the integrity and stability of deep semantic features and providing complete feature support for bidirectional cross-scale semantic transmission. Semantic cache storage mapping formula:

[0192] ;

[0193] in, This represents the global semantic feature data of the original third feature map output by the deep layers of the network; This represents the dedicated full-graph storage address space for semantic caching.

[0194] The three types of cache partitions always maintain independent read / write states, with each partition's read / write bandwidth, address space, and data clock remaining independent, achieving synchronous caching of multi-branch feature data. Combined with a double-buffered ping-pong alternation mechanism, within a single frame inference cycle, the following can be completed simultaneously: writing effective target blocks to the micro-buffer, inputting compressed background blocks into the FIFO, and storing semantic features into the semantic buffer. Simultaneously, the reading and output of the previous frame's cached data are completed in parallel, eliminating data read / write waiting latency. The overall parallel scheduling constraint formula is defined as follows:

[0195] ;

[0196] in, Indicates the total scheduling period of a single frame feature cache; This indicates the time consumed by a single read / write operation of the feature micro-cache; This indicates the time taken for a single queue throughput in the temporary FIFO buffer; This indicates the time taken to store the entire graph in a single semantic cache operation.

[0197] Through this constraint mechanism, synchronous scheduling is completed based on the slowest single partition read / write time, ensuring that the timing of feature data from multiple branches is aligned, with no data misalignment or frame loss.

[0198] After this step is completed, high-value target detail features, lightweight background features, and global semantic features are stored in three independent cache partitions, and a dual-buffered ping-pong read-write mechanism is used to achieve parallel read-write and pipelined scheduling. This invention, through differentiated cache matching and hardware-level parallel scheduling, ensures high-speed access to key target features, reduces the storage overhead of massive background features, and guarantees the complete retention of global semantic features. It effectively solves the problems of limited storage bandwidth, high read-write latency, and severe data congestion on edge chips, providing a high-speed, stable, and time-aligned feature data stream foundation for subsequent cross-scale feature interaction, dynamic weight fusion, and dynamic allocation of computing power.

[0199] This method implements differentiated partitioned storage based on the importance of features, placing high-value, effective target blocks in a high-speed cache to ensure read efficiency; the ping-pong read / write mechanism eliminates read / write waiting latency, and the three types of cache partitions operate independently to avoid data conflicts, achieving efficient scheduling of feature data streams and reducing latency overhead caused by on-chip storage access in edge chips.

[0200] Specifically, a bidirectional cross-scale interaction branch is constructed in the fused convolutional recognition network. Valid target blocks of the first and second filtered features are retrieved from the feature micro-buffer, and compressed background blocks of the first and second filtered features are retrieved from the temporary FIFO. These blocks are then concatenated and recombined according to their spatial positions to obtain complete first and second filtered features. Next, the third feature map is read from the semantic buffer, and the scale cross-correlation matrix of the first, second, and third filtered features is solved using the bidirectional cross-scale interaction branch. Dynamic fusion weights are generated in real time. This includes constructing a bidirectional cross-scale interaction branch, which contains a top-down semantic propagation branch and a bottom-up detail feedback branch. The first branch passes the global semantic information of the third feature map upwards, while the detail feedback branch passes the filtered first and second features downwards. Valid target blocks of the filtered first and second features are retrieved from the feature micro-buffer, and compressed background blocks of the filtered first and second features are retrieved from the temporary FIFO. These blocks are then concatenated and recombined according to their spatial positions to obtain the complete filtered first and second features. The third feature map is then read from the semantic buffer. Matrix operations are used to solve for the spatial and channel correlations between the filtered first and second features and the third feature map. A scale cross-correlation matrix is ​​constructed based on the spatial and channel correlations, and dynamic fusion weights are adaptively allocated based on the correlation values ​​of the scale cross-correlation matrix.

[0201] This application provides an edge image recognition chip acceleration method and system based on multi-scale fusion. The method constructs a bidirectional cross-scale interactive branch including a top-down semantic transfer branch and a bottom-up detail feedback branch. It reads the filtered first feature, filtered second feature and third feature map from the feature micro-buffer, temporary FIFO and semantic buffer. It calculates the spatial correlation and channel correlation of multi-scale features through matrix operations to construct a scale cross-correlation matrix. It adaptively generates dynamic fusion weights based on the correlation degree value within the matrix.

[0202] This invention abandons the traditional one-way top-down feature transfer mode and constructs a dual-path cross-scale interaction branch that includes a top-down semantic transfer branch and a bottom-up detail feedback branch, to achieve bidirectional information compensation and association modeling of three-layer multi-scale features. The branch structure and function are detailed as follows.

[0203] Top-down semantic propagation branch: Using the deep third feature map as the semantic benchmark, high-dimensional global semantic information is propagated upwards to the middle and shallow features step by step, which makes up for the defects of weak semantic information of the first and second features after shallow filtering and easy interference from the background, so as to realize the constraint and correction of local detailed features by global semantics.

[0204] Bottom-up detail feedback branch: Using the features after shallow and medium layer filtering as the detail benchmark, rich edge and texture details are fed down to the deep semantic features step by step, making up for the defects of loss of fine-grained details and weakening of small target features in the third feature map, and realizing the fine supplement of local details to global semantics.

[0205] The bidirectional branch has no additional downsampling or redundant convolutions, and only includes modules for feature alignment, dimension adaptation, correlation calculation, and weight normalization, ensuring low latency and lightweight cross-scale interaction processes, making it suitable for edge chip deployment scenarios.

[0206] Based on the temporal alignment characteristics of the double-buffered ping-pong read-write mechanism described above, this step accurately reads the corresponding feature data from three independent cache partitions to ensure that the features of the input fusion branch are temporally consistent, dimensionally regular, and spatially corresponding. The specific reading logic is as follows:

[0207] Read the first feature after filtering to retain all details Read the second feature after compression and lightweight filtering. Read the third feature map of the complete global semantics. After reading, the three types of features with different resolutions are scale-aligned using the built-in dimension adaptation unit of the branch, unifying the feature channel dimensions and spatial coordinate system, thus eliminating the interference of dimension differences for subsequent cross-scale correlation matrix operations.

[0208] The effective target blocks of the first and second filtered features, each with a size of 16×16, are retrieved from the feature micro-buffer. The 8×8 compressed background payload stored in the temporary FIFO is read out, and the hardware performs a 2x nearest neighbor broadcast upsampling operation in real time to restore the compressed payload to a 16×16 standard background block. The first and second feature maps are then spliced ​​and recombined according to their spatial positions to obtain the complete first and second filtered features.

[0209] Read the restored, sized, and structurally complete first, second, and third feature maps from the feature micro-cache, temporary FIFO, and semantic cache, respectively. Before solving the scale cross-correlation matrix, perform a unified alignment operation for multiple feature dimensions. All features are uniformly adapted to the mid-level feature baseline specification. The specific steps are as follows: Spatial resolution alignment: Use the 208×208 resolution of the second feature map as the unified baseline; use 2×2 average downsampling on the 416×416 first feature map to compress it to 208×208; use nearest neighbor upsampling on the 104×104 third feature map to enlarge it to 208×208, thus completing the complete alignment of the three-layer feature spatial dimensions. Channel Dimension Alignment: For the three layers of features after spatial alignment, independent 1×1 pointwise convolutions are used for channel mapping, uniformly mapping the channel dimension to C=128. A Batch Normalization (BN) layer is then applied after convolution to smooth the feature distribution, ultimately resulting in three layers of standardized features with the same dimension: 208×208 spatial size and 128 channels. These are denoted as standardized shallow features, standardized mid-level features, and standardized deep features, respectively. Alignment Operation Level: All spatial sampling and channel convolution mapping are based on the entire complete feature map, preserving global spatial topology and semantic relationships, and avoiding feature fragmentation problems caused by local block alignment. After dimension alignment, a bidirectional cross-scale interaction branch is used to solve the scale cross-correlation matrix of the three layers of standardized features, and dynamic fusion weights are generated in real time.

[0210] To accurately characterize the degree of information association of features at different scales, this invention simultaneously models the spatial dimension correlation and channel dimension correlation of features, respectively capturing the feature matching degree of the same spatial location and the feature association degree of the same semantic channel, thereby realizing multi-dimensional feature association quantification.

[0211] Spatial correlation is used to characterize the feature similarity of features at the same pixel spatial location across different scales, and measures the degree of matching of local information across scales. The calculation formula is as follows:

[0212] ;

[0213] in, Representing spatial coordinates Spatial correlation coefficient of features at two scales; Represents any two sets of multi-scale features to be associated ( (in pairs) Indicates the feature channel dimension number; This represents the feature magnitude normalization term, used to eliminate the impact of feature magnitude differences on correlation calculation.

[0214] Channel correlation is used to characterize the degree of global association between semantic channels of features at different scales, and measures the complementary matching relationship of semantic information across scales. The calculation formula is as follows:

[0215] ;

[0216] in, Indicates the first Channel and the Channel correlation coefficient; Indicates the corresponding channel number of two sets of features at different scales; Represents the pixel coordinates in the feature space, used for global spatial summation and statistics.

[0217] The correlation matrices for the spatial and channel dimensions are solved separately. Since the two types of matrices are dimensionally incompatible, direct weighted matrix addition is prohibited. Dimensionality is reduced step-by-step to obtain scalar matrices, which are then fused to characterize the degree of correlation among multi-scale features. The spatial correlation matrix is ​​then solved. ,right Perform global mean pooling to obtain the scalar space correlation score. Solve for the channel correlation matrix. ,right Perform global mean pooling to obtain scalar channel correlation scores. The scale cross-correlation matrix is ​​obtained by merging the two types of scalar scores using a weighted fusion of balanced coefficients. The calculation formula is as follows:

[0218] ;

[0219] in, This represents the fusion balance coefficient, which is preferred in this embodiment. It equally considers the correlation characteristics of both spatial and channel dimensions;

[0220] Based on the comprehensive correlation scalar corresponding to each of the three layers of features Softmax normalization is performed to adaptively generate dynamic fusion weights for features at each scale. A complete scale cross-correlation representation system is formed by multiple sets of single-scale comprehensive correlation scalars.

[0221] Unlike traditional fixed fusion weights, this invention uses the correlation coefficients of the scale cross-correlation matrix and adaptively solves the dynamic fusion weights of features at each scale through softmax normalization. Features with higher correlation and greater effective information gain are assigned higher weights, achieving adaptive matching of feature contribution to scenario requirements. The weight calculation formula is as follows:

[0222] ;

[0223] in, The dynamic fusion weights of the k-th scale features satisfy the constraints. ; The scale cross-correlation matrix corresponds to the first The global mean correlation of road features characterizes the overall information effectiveness of features at this scale; This represents the natural exponent operation, used to achieve weight differentiation amplification and normalization; This represents the three features corresponding to the first, second, and third features after filtering, respectively.

[0224] Based on the dynamically calculated fusion weights, the three multi-scale features are weighted and fused to generate a high-quality fused feature map that takes into account shallow details, mid-level transitions, and deep semantics. The complete fusion formula is as follows:

[0225] ;

[0226] in, This represents the final output dynamic fusion feature map; These represent the dynamic adaptive fusion weights for the first, second, and third feature paths, respectively. This represents the three types of multi-scale input features obtained from the cache read.

[0227] Based on three-layer standardized same-dimensional features, the spatial correlation matrix and channel correlation matrix are calculated independently. These two types of matrices have different dimensions and independent physical meanings, and direct matrix addition is prohibited. The computational flow is "independent calculation of dual matrices → feature dimension mapping → feature fusion and normalization": Spatial correlation matrix calculation: For a 208×208 spatial dimension, the pixel spatial similarity of any two scale features is calculated to obtain a global spatial correlation matrix of size 208×208. This is used to characterize the degree of detail matching between features at different scales in the same spatial location. Channel correlation matrix calculation: For a 128-dimensional channel, the channel semantic correlation degree between any two scale features is calculated, resulting in a 128×128 channel correlation matrix. This is used to characterize the degree of semantic complementarity between feature channels at different scales. Dual-matrix compliant fusion mapping: To fuse spatial detail correlation and channel semantic correlation, a unified dimensional mapping is performed on the two types of matrices, avoiding direct matrix addition. Global mean pooling is applied to the spatial correlation matrix, compressing it into a single spatial correlation scalar; channel dimension mean pooling is applied to the channel correlation matrix, compressing it into a single channel correlation scalar; the two scalars are then weighted and fused using preset balance coefficients to obtain the final scale correlation score, achieving dual-dimensional information fusion and avoiding illegal operations on cross-dimensional matrices. The fusion formula is as follows:

[0228] ;

[0229] in, , This is the balance coefficient, with a value range of [0,1] and... Default value Balance coefficient The balance coefficient is used to control the weight of spatial correlation results in the overall correlation score. Used to control the weight of channel-related results in the overall correlation score; This is the spatial correlation matrix; The global mean of the spatial correlation matrix represents the overall spatial matching degree. This is the global mean of the channel correlation matrix, representing the overall channel correlation degree.

[0230] Based on the three-layer normalization features, the correlation scores of each pairwise scale in the three groups—shallow-middle layer, shallow-deep layer, and middle-deep layer—are calculated sequentially. The global mean correlation degree of each feature is obtained through aggregation operations. The specific aggregation rules are as follows: Single feature correlation degree aggregation: Taking the mid-level features as the reference anchor point, the correlation information between each layer feature and the global feature is aggregated respectively. The shallow global correlation degree, mid-level global correlation degree, and deep global correlation degree are obtained from the average of the pairwise correlation scores of the three groups; Normalized weight generation: The global mean correlation degree of the three features is subjected to softmax normalization processing, and the correlation degree value is mapped to a dynamic fusion weight with a sum of 1, as shown in the following formula:

[0231] ;

[0232] in, For the first Global mean correlation of road features This refers to the dynamic fusion weights for the corresponding scale features. The larger the weight value, the higher the correlation between the scale feature and the global features, the greater the proportion of effective information, and the higher its contribution to feature fusion, achieving the effect of adaptively adjusting the fusion weights based on the distribution of image targets.

[0233] This step breaks down the information barrier of unidirectional transmission of multi-scale features by constructing bidirectional cross-scale interactive branches, achieving bidirectional complementarity between detailed and semantic information. It constructs a scale cross-correlation matrix through spatial and channel-dimensional correlation modeling to accurately quantify the information value of features at different scales. Based on matrix correlation, it adaptively generates dynamic fusion weights, achieving intelligent weight allocation under different scenarios and target distributions. Compared to traditional fixed-weight fusion methods, this method can adaptively adapt to various working conditions, such as dense small-target scenes, global large-target scenes, and complex background interference scenes. It retains the detailed features of shallow effective targets while strengthening deep global semantic constraints, providing highly robust and high-information-density fusion feature inputs for subsequent high-precision classification and coordinate regression by the recognition head. Simultaneously, the entire process uses lightweight matrix operations to adapt to the computing power characteristics of edge chips, without additional redundant computing power overhead.

[0234] In this method, bidirectional branches can achieve bidirectional complementarity between deep global semantics and shallow detailed information, making up for the information loss problem in single-direction feature transmission; dynamic fusion weights are generated based on scale cross-correlation matrix, which can adaptively adjust the contribution ratio of each scale feature according to the degree of feature correlation. Unlike fixed weight fusion methods, this method can adapt to the target distribution characteristics in different scenarios and improve the multi-scale feature fusion effect.

[0235] Specifically, when calculating scale information entropy, scale cross-correlation matrix, and fused feature map, a multi-operator parallel hybrid computing approach is used for synchronous computation. Computational resources are dynamically allocated based on the number of effective target blocks, including: setting a preset quantity threshold and reading the number of effective target blocks obtained after the inference of the previous frame as the basis for computing power scheduling in the current frame; in the single-frame inference pipeline, the calculation of scale information entropy and the reorganization of feature selection are completed first, and then the solution of scale cross-correlation matrix and the calculation of fused feature map are executed; when the number of effective target blocks in the previous frame is greater than or equal to the quantity threshold, it is determined to be a target-dense scene, and more computing power is reserved in the current frame, and parallel threads are preferentially allocated to the calculation stage of scale cross-correlation matrix and fused feature map; when the number of effective target blocks in the previous frame is less than the quantity threshold, it is determined to be a target-sparse scene, and more computing power is scheduled in the current frame to the calculation stage of scale information entropy block division and feature selection; among them, a multi-operator parallel hybrid computing approach is used within each stage, and different stages are executed sequentially and step by step according to data dependencies.

[0236] The single-frame inference pipeline exhibits strict data dependencies and is divided into two main sequential computational stages. These stages are executed serially and cannot be synchronized; only within each stage is multi-operator parallel hybrid computation employed. The first stage involves scale information entropy calculation, feature block classification, background compression, and feature reassembly. The second stage can only begin after all computations in this stage are completed. The second stage involves solving the scale correlation matrix, calculating the comprehensive association score, generating dynamic weights, and weighted fusion of multi-scale features. Both stages employ multi-operator synchronous parallel hybrid computation, dynamically allocating parallel computing units and thread resources based on the number of valid target blocks in the previous frame. Each stage independently utilizes multi-operator parallel hybrid computation to improve single-stage computation throughput.

[0237] The system uses preset thresholds for quantity, inter-frame mutation, buffer congestion, and sliding filter window length. Computational power scheduling is based on a multi-condition composite judgment, no longer solely relying on the number of valid target blocks in a single original frame. It also incorporates several boundary control logics: First-frame initialization control: Since the first frame of the video stream has no historical statistical data, balanced default computational power allocation is enabled, and computational power weights are adjusted during the entropy filtering phase. Computing power weight in the fusion computing stage After the inference of the first frame is completed, the number of valid target blocks is counted and stored in a register as the basis for subsequent frame scheduling; Scene change smoothing control: calculate the absolute value of the difference between the number of valid blocks in the current frame and the previous frame. If the difference exceeds the inter-frame mutation threshold, a scene mutation is determined, and the current computing power weight is locked for three consecutive frames to prevent inference jitter caused by sudden and large-scale switching of computing power resources; Cache congestion fallback control: Real-time monitoring of feature micro-cache, temporary FIFO, and semantic cache occupancy rates. When any cache occupancy rate exceeds the congestion threshold, the computing power weight in the entropy filtering stage is forcibly increased, prioritizing background block compression and redundant data cleanup to reduce cache storage load; Threshold oscillation suppression control: A sliding window mean filter is applied to the number of effective target blocks in the previous three consecutive frames, using the filtered mean... Instead of comparing the original single-frame values ​​with a preset threshold, it suppresses frequent switching of computing power weights caused by minor fluctuations in the image; the standard scheduling logic is: when there is no first frame, no scene changes, and no buffer congestion, the average number of effective target blocks after sliding window filtering is used. Differentiate between dense and sparse target scenarios and dynamically allocate parallel computing resources for the two computation stages.

[0238] After each frame of image inference is completed, the system automatically counts and caches the total number of valid target blocks obtained in the current frame, denoted as . This serves as the sole priori basis for computing power scheduling in the next frame, completely avoiding the logical flaw of "no statistics available before the current frame is fully computed." A preset threshold for the number of judgments in a fixed scenario is established. This threshold is fixedly configured based on the edge chip's computing power specifications, adapting to the upper limit of embedded hardware computing power. Scenario determination rules:

[0239] ;

[0240] in, This is the total number of valid target blocks counted after the inference of the previous frame is completed. It is an integer and represents the density of targets in the image. Set a preset quantity threshold and fix the hyperparameters. Used to distinguish between dense and sparse scenes.

[0241] Based on the above determination results, the parallel computing power weights of the two major computing stages in the current frame are dynamically allocated, and the computing power weight of the entropy stage is defined. Computing power weight in the integration phase The weights are always 1, representing the allocation ratio of hardware threads, computing units, and scheduling priorities. Computing power allocation formula:

[0242] ;

[0243] in, The computational power is allocated weights for the scale information entropy calculation and feature selection stages. The higher the weight, the more parallel threads are allocated and the higher the computation priority. This mechanism allocates computational power weights for the scale cross-correlation matrix, dynamic weight generation, and fusion feature map calculation stages. Mechanism explanation: In target-dense scenarios, fusion matrix operations and feature weighted fusion are computational bottlenecks, so computational power is prioritized for the fusion stage; in target-sparse scenarios, a large number of redundant background blocks require entropy filtering and compression, so computational power is prioritized for the entropy calculation and filtering stage, accurately matching the computational power consumption characteristics of different scenarios.

[0244] Scale information entropy and feature selection and reorganization stage: This stage is a necessary pre-stage of the inference pipeline. All subsequent fusion calculations depend on the output of this stage and cannot be executed synchronously with the fusion stage. Internally, this stage supports parallel entropy calculation for multiple feature blocks and parallel selection for multiple regions, achieving intra-stage parallel acceleration. Unified formula for calculating scale information entropy:

[0245] ;

[0246] in, The scale information entropy of a single 16×16 non-overlapping feature block represents the effective richness of the features; This represents the total number of statistical categories of pixel grayscale and texture gradient within the feature block; For the first The distribution probability of pixel / texture-like features satisfies Three-class block determination formula:

[0247] ;

[0248] in, The first entropy threshold is used to determine the upper threshold for effective targets. This is the second entropy threshold, the threshold for determining invalid background.

[0249] Stage-based parallelism rule: Multiple operators, including entropy calculation for all feature blocks, threshold determination, background block downsampling and compression, zero-padding of invalid blocks, and feature space reorganization, are executed in parallel, relying on the above-assigned... The computing power is used to schedule parallel threads to complete all block processing in batches and output the complete first and second features after filtering.

[0250] Cross-scale correlation matrix and fusion feature calculation stage: This stage is a post-dependent stage and must wait for the first stage of feature selection and reorganization to complete before it can start; internally, it implements parallel multi-operator operations for dimension alignment, spatial correlation calculation, channel correlation calculation, weight normalization, and feature weighted fusion, relying on... Allocate computing power resources to accelerate inference.

[0251] Feature dimension alignment (pre-unification):

[0252] Based on the 208×208 mid-layer features, shallow features are downsampled and deep features are upsampled. Combined with 1×1 convolution with a unified channel number of 128, three layers of standardized features of the same dimension are obtained, providing a dimensional basis for matrix operations.

[0253] Spatial correlation matrix calculation (parallel pixel-level computation):

[0254] ;

[0255] in, This is a spatial correlation matrix with a size of 208×208, representing the degree of matching of features at the same spatial location at different scales; For any two groups of aligned, standardized multiscale features.

[0256] Channel correlation matrix calculation (parallel channel-level operations):

[0257] ;

[0258] in, This is the channel correlation matrix, with a size of 128×128, representing the semantic correlation degree of the channels; For the first 3D channel feature vectors.

[0259] Dual-matrix compliance fusion score (solving the problem of non-additivity between different dimensions):

[0260] ;

[0261] in, This is the space and channel balance coefficient. The default is 0.5; mean(M_s) is the global mean of the spatial matrix. The global mean of the channel matrix; It is the comprehensive correlation score of pairwise scale features.

[0262] Dynamic fusion weight normalization:

[0263] ;

[0264] in, For the first Global mean correlation of road-scale features; For the first The road features are finally dynamically fused into weights, and the sum of the three road weights is 1.

[0265] Formula for generating feature fusion map:

[0266] ;

[0267] in, For the final fused feature map; For the first Normalized multi-scale features after path alignment. After all inference for the current frame is completed, the number of valid target blocks in the current frame is recounted. Overwrite cache This serves as the basis for scheduling the next frame, enabling adaptive computing power iterative scheduling between frames of continuous video streams.

[0268] This method relies on the scene characteristics of the number of effective target blocks to dynamically tilt the priority of computing power. In the scene with dense targets, it ensures the accuracy of multi-scale feature fusion operation, and in the scene with sparse targets, it enhances the efficiency of feature selection. Combined with multi-operator parallel computing, it gives full play to the computing unit capabilities of edge chips, realizes on-demand scheduling of computing resources, and reduces the power consumption and latency overhead caused by invalid operations.

[0269] Specifically, the fused feature map is input into the recognition head of the fused convolutional recognition network, and the recognition head is used to classify and detect the target in the fused feature map to obtain the target category and target coordinates. This includes: inputting the fused feature map into the recognition head of the fused convolutional recognition network, wherein the recognition head is a two-branch detection and classification structure, which includes a grid classification branch and a grid regression branch.

[0270] The grid classification branch is used to perform grid-based local feature aggregation on the fused feature map. The target class probability of each grid is output through convolutional mapping to obtain the multi-target class distribution. Multiple sets of anchor boxes are preset. Based on the preset sets of anchor boxes, the grid regression branch is used to calculate the four-dimensional coordinate offset of the effective target region in the fused feature map grid by grid. Combined with the grid position and the anchor box reference, the independent coordinate information of multiple targets is decoded. Finally, the target class and target coordinate set corresponding to all effective targets in the fused feature map are output.

[0271] Feature Map Input and Grid Partition Initialization: The fused feature map, after multi-scale weighted fusion, is input into the recognition head of the fused convolutional recognition network. The recognition head has a built-in parallel structure of grid classification and grid regression branches, and global pooling is not performed throughout the process, fully preserving the spatial, positional, and channel dimensions of the feature map. Based on the resolution of the fused feature map, a uniform, non-overlapping prediction grid is divided. Each grid corresponds to a fixed region of the original image and serves as an independent prediction unit, enabling one grid for one independent target prediction, providing a structural foundation for multi-target parallel detection.

[0272] Grid Classification Branch – Grid-by-Grid Local Feature Aggregation and Category Probability Prediction: This branch is responsible for multi-target category recognition. It abandons the global feature aggregation method and adopts a dense prediction mode of grid-by-grid local feature aggregation + convolutional mapping. The specific process is as follows: Local Feature Aggregation: For each preset grid region, extract local fusion features within the grid range, aggregate spatial detail features and multi-scale semantic features within the grid, weaken irrelevant global background interference, and focus on local target feature information within the grid; Convolutional Feature Mapping: Use multi-layer 1×1 convolutions to map the channel dimensions and purify the features of the aggregated grid features, without changing the feature space size and preserving grid position information; Category Probability Output: Map the convolution output features to normalized probabilities for each category using the Softmax activation function, obtaining an independent target category probability vector for each grid; Global Category Distribution Generation: Traverse the prediction results of all grids, concatenate all grid category probabilities to form the multi-target category distribution of the entire feature map, completing the initial category determination of all potential targets in the image.

[0273] After convolution mapping of the single-grid, single-anchor-box features, Softmax normalization is used to obtain the class probability distribution, as shown in the following formula:

[0274] ;

[0275] in, For feature map grid coordinates, For row number, For column numbers; The anchor frame number corresponding to the current grid; The target category number; Total number of categories; The raw (unnormalized) class scores output by the network convolution. The normalized categories corresponding to the current grid and the current anchor frame. The predicted probability, with a value range of [0,1].

[0276] Formula for determining the highest probability category:

[0277] ;

[0278] in, The optimal target category label predicted for the current grid and the current anchor box.

[0279] Multiple sets of anchor boxes are pre-configured for grid target matching initialization: To adapt to the multi-target detection requirements of different scales and aspect ratios, multiple sets of benchmark anchor boxes are pre-configured. The anchor box parameters are obtained based on the clustering of target sizes in the dataset, including small, medium, and large scale anchor boxes, covering all common target sizes in the image. All anchor boxes are bound to grid positions one by one, and each prediction grid is equipped with multiple sets of anchor boxes, realizing multi-scale target matching in a single grid. This solves the problem that a single anchor box cannot adapt to multiple scales and multiple targets, and provides a benchmark for subsequent coordinate regression decoding.

[0280] Grid Regression Branch – Calculating Grid-by-Grid Coordinate Offsets: The regression branch operates in parallel with the classification branch. Based on local features of the same grid and using multiple pre-set anchor boxes as a reference, it calculates the four-dimensional coordinate offset grid-by-grid and anchor box-by-anchor box. The four-dimensional parameters correspond to: center x-coordinate offset, center y-coordinate offset, width scaling offset, and height scaling offset. The branch only generates valid offset parameters for regions containing valid targets within the grid, outputting offsets close to 0 for pure background grids. This achieves accurate regression of the target region and ineffective suppression of the background region, and batch outputs the set of coordinate offsets corresponding to all grids and all anchor boxes.

[0281] The network predicts the four-dimensional offset relative to the preset anchor frame, corresponding to the center coordinates and the width and height scaling offsets, as shown in the following formula:

[0282] ;

[0283] in, The horizontal and vertical offsets of the target box center coordinates relative to the anchor box; The logarithmic scaling offset of the target bounding box's width and height relative to the anchor box; Set the center coordinates, width, and height of the preset anchor frame; The coordinates of the original predicted bounding box in the network.

[0284] Multi-target coordinate decoding generates independent target coordinate information: Taking grid spatial position, preset anchor frame reference parameters, and grid regression offset as input, the coordinates of each grid and each set of anchor frames are decoded one by one, correcting the anchor frame reference position and size deviation, to obtain accurate pixel coordinate frames of multiple independent targets. The decoding logic strictly follows the topological relationship of the feature map space, ensuring that each decoded coordinate frame corresponds to the real target area in the image, and outputs multiple sets of target coordinate information that do not interfere with each other and correspond independently, completely breaking through the limitation of single target coordinate output.

[0285] Based on grid positions, anchor frame references, and predicted offsets, reverse decoding yields the true pixel coordinate boxes of the image, forming the core formula for multi-object detection:

[0286] ;

[0287] in, The pixel coordinates and dimensions of the real target detection box obtained after decoding are used; the offset is corrected by weighting the anchor box size to eliminate grid scale differences and ensure consistent decoding accuracy for multi-scale targets.

[0288] Effective target selection and multi-target result set output: Confidence screening: Based on the probability of each grid category, effective targets with probabilities higher than a preset confidence threshold are selected, eliminating background misjudgments and low-confidence invalid predictions; Redundant box deduplication: Optional edge-lightweight NMS non-maximum suppression algorithm is used to deduplicate multiple overlapping prediction boxes of the same target, retaining the optimal accurate target box; Result aggregation: The category labels and corresponding accurate coordinate boxes of all effective targets are integrated to generate a multi-target category-coordinate mapping set as the final recognition output result. The formula for eliminating low-confidence false detections and retaining effective targets is as follows:

[0289] ;

[0290] in, The maximum class probability for a single grid and single anchor box; Preset a reliability threshold (fixed hyperparameter); This is the effective target mask, where 1 represents an effective target and 0 represents an invalid background prediction.

[0291] For multiple overlapping predicted bounding boxes with the same target, the optimal box is selected by intersection-union ratio (IoU) to adapt to lightweight inference at the edge:

[0292] ;

[0293] when If the boxes are identified as redundant boxes for the same target, the box with the highest confidence level is retained, and the remaining overlapping boxes are removed. For two overlapping predicted bounding boxes; The area of ​​the bounding box in pixels; This is a preset intersection-union ratio threshold used for redundant bounding boxes.

[0294] The final output is a set of binary pairs representing valid targets, defined by the formula:

[0295] ;

[0296] in, The valid target number; For the first One target category; For the first The actual coordinates and dimensions of each target; This represents the confidence probability of the corresponding target.

[0297] This method employs a dual-branch parallel inference architecture to separate category recognition and location calculation, each performing its own function. By relying on multi-scale information in the fused feature map that takes into account both details and semantics, inference can be carried out simultaneously to ensure the accuracy of target classification and the precision of localization, and efficiently output the category and location information required for recognition to complete the image target detection task.

[0298] This system is equipped with a customized edge recognition chip hardware architecture. The overall hardware consists of an image input path, a multi-operator processing unit array, multiple independent cache banks with descriptor caches, a global computing power scheduler, a detection output processing unit, and a separate data / control dual bus forming a complete closed-loop interconnection path. Among them, the feature micro-cache bank, the temporary FIFO cache bank, and the semantic cache bank are divided into mutually isolated dedicated on-chip address spaces. Each of the three types of cache banks is equipped with an independent parallel read / write data port. The temporary FIFO cache bank integrates a dedicated descriptor cache for synchronously storing the block row number, column number, and block type topology index corresponding to each compressed background payload. Double-buffered ping-pong read... The writing process is implemented using two sets of sub-buffer units built into the cache bank, combined with time-division multiplexing dual-port hardware, enabling physical parallel completion of feature writing and reading operations. The multi-operator processing unit array is further subdivided into entropy calculation operator sub-arrays, spatial and channel correlation matrix operator sub-arrays, and feature weighted fusion operator sub-arrays. Each operator sub-array is configured with an independent computing port. The global computing power scheduler connects all operator sub-arrays and the control ports of the three types of cache banks simultaneously through a dedicated control bus. The scheduler has built-in frame valid block statistics registers, sliding window filtering circuits, scene change comparison circuits, and cache congestion judgment circuits, and can collect the number of valid target blocks output by the descriptor cache and the address space of each cache bank in real time. The system uses hardware feedback signals for occupancy rates and outputs time-sharing signals and parallel resource allocation signals based on built-in multi-constraint logic. This strictly controls the serial execution order of the entropy calculation stage and the fusion calculation stage, while allocating corresponding parallel computing resources to the operators within each stage. In terms of hardware data flow, the image acquisition module connects to the input end of the multi-operator processing unit array via a high-speed input port. The block features processed by the array are distributed to the write ports of three types of cache banks via independent distribution ports. During cross-scale fusion calculation, the cache banks output data through dedicated read ports. Based on the topological index of the descriptor cache, the compressed features are upsampled and restored in real time and then sent to the relevant matrix operator subarray to complete the correlation. The computation and dynamic weight generation, and the weighted fusion feature map are directly connected to the grid classification and grid regression operation units corresponding to the dual-branch detection head via a dedicated output port. The cache bank partitioning rules, port allocation logic, descriptor storage mechanism, processing unit array partitioning method, and scheduler linkage control circuit of the entire hardware are all tailored to the multi-scale entropy screening, hierarchical compression storage, topology space restoration, dimension-compatible feature fusion, and multi-constraint adaptive computing power scheduling methods of this application. Each hardware unit has fixed and unique interconnection constraints and signal interaction logic, which cannot be directly applied to general CNN accelerator hardware, forming a collaborative whole in which the algorithm process and customized hardware are deeply bound.

[0299] Figure 2 A connection diagram of an edge image recognition chip acceleration system based on multi-scale fusion provided in this application is shown below. Figure 2As shown, this embodiment provides an edge image recognition chip acceleration system based on multi-scale fusion. The acceleration system includes:

[0300] The image acquisition and feature extraction module is used to acquire raw image data of the target area using an image sensor, normalize and scale the raw image data to obtain standard image data, construct a fusion convolutional recognition network, and input the standard image data into the fusion convolutional recognition network for forward inference to obtain multi-scale raw features, which include a first feature map, a second feature map and a third feature map.

[0301] The feature block entropy filtering and recombination module is connected to the image acquisition and feature extraction module. The feature block entropy filtering and recombination module is used to preset size threshold and entropy threshold, divide the first feature map and the second feature map into multiple non-overlapping feature blocks according to the size threshold, calculate the scale information entropy of each non-overlapping feature block, compare multiple scale information entropy with entropy threshold to obtain effective target blocks, smooth background blocks and blank invalid blocks, sample and compress the smooth background blocks to obtain compressed background blocks, and discard blank invalid blocks, recombine the effective target blocks and compressed background blocks in the first feature map to obtain the filtered first feature, and recombine the effective target blocks and compressed background blocks in the second feature map to obtain the filtered second feature.

[0302] The hierarchical cache parallel scheduling module is connected to the feature block entropy filtering and reorganization module. The hierarchical cache parallel scheduling module is used to write the effective target blocks in the first and second filtered features into the feature micro-cache using a double-buffered ping-pong read-write mechanism, write the compressed background blocks in the first and second filtered features into a temporary FIFO, and store the third feature map into the semantic cache. The double-buffered ping-pong read-write mechanism can perform data writing and data reading operations in parallel.

[0303] The dynamic fusion weight module is connected to the hierarchical cache parallel scheduling module. The dynamic fusion weight module is used to construct a bidirectional cross-scale interactive branch in the fusion convolutional recognition network. It reads the filtered first feature, filtered second feature and third feature map from the feature micro-cache, temporary FIFO and third feature map respectively. It uses the bidirectional cross-scale interactive branch to solve the scale cross-correlation matrix of the filtered first feature, filtered second feature and third feature map and generates dynamic fusion weights in real time.

[0304] The fusion feature map construction module is connected to the dynamic fusion weight module. The fusion feature map construction module is used to perform weighted fusion of the first feature, the second feature and the third feature map after screening based on the dynamic fusion weight to obtain the fusion feature map. When calculating the scale information entropy, the scale cross-correlation matrix and the fusion feature map, a multi-operator parallel hybrid computing method is used for synchronous operation, and computing resources are dynamically allocated according to the number of effective target blocks.

[0305] The recognition head inference output module is connected to the fusion feature map construction module. The recognition head inference output module is used to input the fusion feature map into the recognition head of the fusion convolutional recognition network, and use the recognition head to classify and detect the target in the fusion feature map to obtain the target category and target coordinates.

[0306] This application provides an edge image recognition chip acceleration system based on multi-scale fusion. The system sequentially includes an image acquisition and feature extraction module, a feature block entropy filtering and reorganization module, a hierarchical cache parallel scheduling module, a dynamic fusion weight module, a fusion feature map construction module, and a recognition head inference output module. The image acquisition and feature extraction module acquires and preprocesses image data, and outputs three layers of multi-scale original features through a fusion convolutional recognition network. The feature block entropy filtering and reorganization module completes shallow and medium-level feature block determination, background compression, and feature reorganization based on scale information entropy. The hierarchical cache parallel scheduling module realizes partitioned storage of different types of features and supports parallel reading and writing by using a double-buffered ping-pong read-write mechanism. The dynamic fusion weight module constructs a bidirectional cross-scale interactive branch to read cached features, solves the scale cross-correlation matrix to generate dynamic fusion weights, and obtains fusion feature maps by cooperating with multi-operator parallel operations and dynamic computing power scheduling. Finally, the recognition head inference output module completes target classification and coordinate detection.

[0307] The system adopts a modular serial interconnect architecture with clear functional division of each module and orderly data flow. It reduces data redundancy through feature entropy screening, improves data throughput efficiency through differentiated hierarchical caching, and combines bidirectional cross-scale dynamic weight fusion and adaptive computing power scheduling to fully adapt to the hardware characteristics of edge chips with limited storage and computing power. It can reduce inference latency and power consumption while ensuring recognition accuracy, and achieve efficient acceleration of image recognition tasks on embedded devices.

Claims

1. A method for accelerating edge image recognition chips based on multi-scale fusion, characterized in that, The acceleration method includes: The original image data of the target area is acquired using an image sensor. The original image data is normalized and scaled to obtain standard image data. A fusion convolutional recognition network is constructed. The standard image data is input into the fusion convolutional recognition network for forward inference to obtain multi-scale original features. The multi-scale original features include a first feature map, a second feature map, and a third feature map. A preset size threshold and an entropy threshold are used to divide the first feature map and the second feature map into multiple non-overlapping feature blocks according to the size threshold. The scale information entropy of each non-overlapping feature block is calculated. Multiple scale information entropies are compared with the entropy threshold to obtain effective target blocks, smooth background blocks, and blank invalid blocks. The smooth background blocks are sampled and compressed to obtain compressed background blocks, and the blank invalid blocks are discarded. The effective target blocks in the first feature map and the compressed background blocks are recombined to obtain the filtered first feature. The effective target blocks in the second feature map and the compressed background blocks are recombined to obtain the filtered second feature. The effective target blocks in the first and second filtered features are written into the feature micro-buffer using a double-buffered ping-pong read-write mechanism, the compressed background blocks in the first and second filtered features are written into a temporary FIFO, and the third feature map is stored in the semantic cache. The double-buffered ping-pong read-write mechanism can perform data writing and data reading operations in parallel. In the fused convolutional recognition network, a bidirectional cross-scale interactive branch is constructed. The effective target blocks of the first filtered feature and the second filtered feature are retrieved from the feature micro-buffer, and the compressed background blocks of the first filtered feature and the second filtered feature are retrieved from the temporary FIFO. These blocks are then concatenated and recombined according to their spatial positions to obtain the complete first filtered feature and the second filtered feature. Next, the third feature map is read from the semantic buffer, and the scale cross-correlation matrix of the first filtered feature, the second filtered feature, and the third feature map is solved using the bidirectional cross-scale interactive branch. Dynamic fusion weights are generated in real time. Based on the dynamic fusion weight, the first filtered feature, the second filtered feature, and the third feature map are weighted and fused to obtain a fused feature map. When calculating the scale information entropy, the scale cross-correlation matrix, and the fused feature map, a multi-operator parallel hybrid computing method is used for synchronous operation, and computing resources are dynamically allocated according to the number of effective target blocks. The fused feature map is input into the recognition head of the fused convolutional recognition network, and the recognition head is used to classify and detect the target in the fused feature map to obtain the target category and target coordinates.

2. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, The process of acquiring raw image data of the target area using an image sensor, and then normalizing and scaling the raw image data to obtain standard image data includes: A preset resolution and frame rate are used to acquire the original image data of the target area using the image sensor based on the preset resolution and frame rate. The pixel values ​​of the original image data are uniformly mapped to the standard range of [0,1] to obtain normalized image data; A standard size is preset, and the normalized image data is scaled to the standard size using a bilinear interpolation algorithm while preserving edge details and texture features to obtain the standard image data.

3. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, The step of inputting the standard image data into the fused convolutional recognition network for forward inference to obtain multi-scale original features includes: First, the first feature map in the standard image data is extracted using the ordinary convolutional layer in the fused convolutional recognition network. Secondly, the separable convolutional layer in the fused convolutional recognition network is used to extract the second feature map from the standard image data; Finally, the dilated convolutional layer in the fused convolutional recognition network is used to extract the third feature map from the standard image data.

4. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, The step of dividing the first feature map and the second feature map into multiple non-overlapping feature blocks according to the size threshold, and calculating the scale information entropy of each non-overlapping feature block, includes: A preset size threshold is used to divide the first feature map and the second feature map into multiple non-overlapping feature blocks based on the feature map resolution of the first feature map and the second feature map and the size threshold. The scale information entropy of each non-overlapping feature block is calculated by traversing multiple non-overlapping feature blocks and employing a channel-by-channel independent calculation strategy.

5. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, The process of comparing multiple scale information entropies with the entropy threshold to obtain effective target blocks, smooth background blocks, and blank invalid blocks, sampling and compressing the smooth background blocks to obtain compressed background blocks, and discarding the blank invalid blocks, recombining the effective target blocks and compressed background blocks in the first feature map to obtain a filtered first feature, and recombining the effective target blocks and compressed background blocks in the second feature map to obtain a filtered second feature, includes: A first entropy threshold and a second entropy threshold are preset. Non-overlapping feature blocks with scale information entropy greater than or equal to the first entropy threshold are determined as valid target blocks; non-overlapping feature blocks with scale information entropy greater than or equal to the second entropy threshold and less than the first entropy threshold are determined as smooth background blocks; and non-overlapping feature blocks with scale information entropy less than the second entropy threshold are determined as blank invalid blocks. The smooth background block is sampled and compressed using a mean downsampling compression algorithm to obtain the compressed background block, while discarding the blank invalid block; Based on the spatial relationship between the first feature map and the second feature map, the effective target block and the compressed background block are recombined to obtain the filtered first feature and the filtered second feature, wherein the filtered first feature and the filtered second feature are consistent with the feature size and spatial structure of the first feature map and the second feature map, respectively.

6. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, The process of using a double-buffered ping-pong read / write mechanism to write the effective target blocks from the filtered first feature and the filtered second feature into a feature micro-buffer, writing the compressed background blocks from the filtered first feature and the filtered second feature into a temporary FIFO, and storing the third feature map into a semantic buffer includes: The effective target blocks in the first and second filtered features are stored in the feature micro-cache with small capacity and fast read / write speed using a double-buffered ping-pong read / write mechanism. The compressed background block from the first filtered feature and the second filtered feature is written into the temporary FIFO using the double-buffered ping-pong read / write mechanism. The third feature map is stored in the semantic cache using the double-buffered ping-pong read / write mechanism. The dual-buffered ping-pong read / write mechanism includes a first buffer unit and a second buffer unit, wherein the first buffer unit and the second buffer unit alternately perform the data write operation and the data read operation; The feature micro-cache, the temporary FIFO, and the semantic cache are read and written independently and do not interfere with each other.

7. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, In the fused convolutional recognition network, a bidirectional cross-scale interactive branch is constructed. The effective target blocks of the first filtered feature and the second filtered feature are respectively retrieved from the feature micro-buffer. The compressed background blocks of the first filtered feature and the second filtered feature are retrieved from the temporary FIFO. They are then spliced ​​and recombined according to the spatial positions of the first feature map and the second feature map to obtain the complete first filtered feature and the second filtered feature. Then, the third feature map is read from the semantic cache, and the scale cross-correlation matrix of the filtered first feature, the filtered second feature, and the third feature map is solved using the bidirectional cross-scale interaction branch. Dynamic fusion weights are generated in real time, including: Construct the bidirectional cross-scale interaction branch, wherein the bidirectional cross-scale interaction branch includes a top-down semantic transmission branch and a bottom-up detail feedback branch. The semantic transmission branch transmits the global semantic information of the third feature map upward, and the detail feedback branch transmits the filtered first feature and the filtered second feature downward. The effective target blocks of the filtered first feature and the filtered second feature are retrieved from the feature micro-cache, respectively. The compressed background blocks of the filtered first feature and the filtered second feature are retrieved from the temporary FIFO. They are then spliced ​​and recombined according to the spatial positions of the first feature map and the second feature map to obtain the complete filtered first feature and the filtered second feature. The third feature map is then read from the semantic cache. The spatial and channel correlations of the first, second, and third feature maps after filtering are solved by matrix operations. The scale cross-correlation matrix is ​​constructed based on the spatial and channel correlations. The dynamic fusion weights are adaptively allocated based on the correlation degree values ​​of the scale cross-correlation matrix.

8. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, When calculating the scale information entropy, the scale cross-correlation matrix, and the fused feature map, a multi-operator parallel hybrid computing method is used for synchronous computation, and computing resources are dynamically allocated according to the number of effective target blocks, including: A preset quantity threshold is used to determine the number of valid target blocks obtained after the inference of the previous frame, which is then used as the basis for computing power scheduling in the current frame. In the single-frame inference pipeline, the calculation of the scale information entropy and the reorganization of the feature selection are completed first, and then the solution of the scale cross-correlation matrix and the calculation of the fused feature map are performed. When the number of effective target blocks in the previous frame is greater than or equal to the number threshold, it is determined to be a target-dense scene. In the current frame, more computing power is reserved and parallel threads are preferentially allocated to the calculation stage of the scale cross-correlation matrix and the fused feature map. When the number of valid target blocks in the previous frame is less than the number threshold, it is determined to be a target sparse scene. In the current frame, more computing power will be allocated to the calculation of the scale information entropy block and the feature selection stage. Among them, the multi-operator parallel hybrid operation method is adopted within each stage, and different stages are executed sequentially according to the data dependency relationship.

9. The edge image recognition chip acceleration method based on multi-scale fusion according to claim 1, characterized in that, The step of inputting the fused feature map into the recognition head of the fused convolutional recognition network, and using the recognition head to classify and detect targets in the fused feature map to obtain target category and target coordinates, includes: The fused feature map is input into the recognition head of the fused convolutional recognition network, wherein the recognition head is a dual-branch detection and classification structure, and the dual-branch detection and classification structure includes a grid classification branch and a grid regression branch; The fused feature map is aggregated using the grid classification branch, and the target class probability of each grid is output through convolutional mapping to obtain the multi-target class distribution. Multiple sets of anchor frames are preset. Based on these preset sets of anchor frames, the four-dimensional coordinate offset of the effective target region in the fused feature map is calculated grid by grid using the grid regression branch. Combining the grid position and the anchor frame reference, the independent coordinate information of multiple targets is decoded. Finally, the target category and target coordinate set corresponding to all effective targets in the fused feature map are output.

10. An edge-end image recognition chip acceleration system based on multi-scale fusion, characterized in that, The acceleration system is applied to the acceleration method according to any one of claims 1 to 9, and the acceleration system comprises: The image acquisition and feature extraction module is used to acquire raw image data of the target area using an image sensor, normalize and scale the raw image data to obtain standard image data, construct a fusion convolutional recognition network, and input the standard image data into the fusion convolutional recognition network for forward inference to obtain multi-scale raw features, wherein the multi-scale raw features include a first feature map, a second feature map and a third feature map; The feature block entropy filtering and recombination module is connected to the image acquisition and feature extraction module. The feature block entropy filtering and recombination module is used to preset a size threshold and an entropy threshold, divide the first feature map and the second feature map into multiple non-overlapping feature blocks according to the size threshold, calculate the scale information entropy of each non-overlapping feature block, compare multiple scale information entropies with the entropy threshold to obtain effective target blocks, smooth background blocks and blank invalid blocks, sample and compress the smooth background blocks to obtain compressed background blocks, and discard the blank invalid blocks, recombine the effective target blocks in the first feature map with the compressed background blocks to obtain the filtered first feature, and recombine the effective target blocks in the second feature map with the compressed background blocks to obtain the filtered second feature. A hierarchical cache parallel scheduling module is connected to the feature block entropy filtering and reorganization module. The hierarchical cache parallel scheduling module is used to write the effective target blocks in the first and second filtered features into the feature micro-cache using a double-buffered ping-pong read-write mechanism, write the compressed background blocks in the first and second filtered features into a temporary FIFO, and store the third feature map into a semantic cache. The double-buffered ping-pong read-write mechanism can perform data writing and data reading operations in parallel. The dynamic fusion weight module is connected to the hierarchical cache parallel scheduling module. The dynamic fusion weight module is used to construct a bidirectional cross-scale interaction branch in the fused convolutional recognition network. It reads the filtered first feature, the filtered second feature, and the third feature map from the feature micro-cache, the temporary FIFO, and the semantic cache, respectively. It uses the bidirectional cross-scale interaction branch to solve the scale cross-correlation matrix of the filtered first feature, the filtered second feature, and the third feature map, and generates dynamic fusion weights in real time. A fusion feature map construction module is connected to the dynamic fusion weight module. The fusion feature map construction module is used to perform weighted fusion of the first filtered feature, the second filtered feature and the third feature map based on the dynamic fusion weight to obtain a fusion feature map. When calculating the scale information entropy, the scale cross-correlation matrix and the fusion feature map, a multi-operator parallel hybrid computing method is used for synchronous operation, and computing resources are dynamically allocated according to the number of effective target blocks. The recognition head inference output module is connected to the fusion feature map construction module. The recognition head inference output module is used to input the fusion feature map into the recognition head of the fusion convolutional recognition network, and use the recognition head to classify and detect the fusion feature map to obtain the target category and target coordinates.

Images