Image processing method, system and storage medium based on local binary convolution

Abstract

This application relates to an image processing method, system, and storage medium based on local binary convolution. It includes: acquiring the original image to be processed; performing feature-based reprocessing on the original image to obtain an initial feature map; constructing a local binary convolution model based on a binary neural network; binarizing the initial feature map using the binary convolution model to obtain a binarized feature map; using the center pixel as a local threshold for neighboring pixels in the binarized feature map and collecting high-order image statistical information; and realizing image processing through local binary convolution by having the binary neural network recognize the high-order image statistical information. This application aims to improve the accuracy of BNN structures. This method can effectively capture high-order image statistical information with maximum information retention from the input feature map through a local thresholding strategy, thereby enhancing the representation and discrimination capabilities of the BNN and improving the accuracy of image recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image processing method, system, and storage medium based on local binary convolution. Background Technology

[0002] A major research direction in artificial intelligence (AI) is the development of increasingly massive and computationally expensive deep learning models. These models focus on improving predictive accuracy by using massive amounts of computing power and data, while neglecting other important aspects such as sustainability, environmental friendliness, and deployment on resource-constrained edge devices. Many researchers have recognized this and advocate for increased research into green AI to achieve significant reductions in computational costs or to address scenarios with limited resources without significantly sacrificing model accuracy.

[0003] In recent years, researchers have made numerous efforts to obtain deep models with low size, low latency, and low power consumption, such as network pruning, compact and lightweight network design, and network quantization. Among these efforts, binary neural networks (BNNs), where weights (and possibly activation values) are restricted to binary values ​​(i.e., +1 and -1) rather than full-precision floating-point numbers, appear to be an attractive technique for deploying deep models on resource-constrained edge devices due to their hardware friendliness, memory efficiency, and computational efficiency.

[0004] For BNNs with both binary weights and activations, the following challenges exist: gradient mismatch between forward and backward propagation during training due to the non-differentiable Sign binary function; large quantization errors between full-precision activations and binary activations (or latent weights and binary weights during training); and inevitably severe information loss due to limited representational power caused by extreme quantization. All of these contribute to the accuracy gap when performing image processing based on BNNs. Summary of the Invention

[0005] Therefore, it is necessary to provide an image processing method, system, device, and storage medium based on local binary convolution to address the aforementioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide an image processing method based on local binary convolution, the method comprising:

[0007] Obtain the original image to be processed, and perform feature reprocessing on the original image to obtain an initial feature map;

[0008] A local binary convolution model is constructed based on a binary neural network. The initial feature map is then binarized using the binary convolution model to obtain a binarized feature map.

[0009] The neighboring pixels in the binarized feature map are used with the center pixel as the local threshold, and high-order image statistical information is collected.

[0010] Image processing with local binary convolution is achieved by having a binary neural network recognize the higher-order image statistical information.

[0011] Furthermore, the step of constructing a local binary convolution model based on a binary neural network, and then binarizing the initial feature map using the binary convolution model to obtain a binarized feature map, includes:

[0012] An adaptive kernel is obtained by modifying the identity kernel in the original binary convolution to capture and retain the high-order information of the initial feature map during the preset binarization process;

[0013] An auxiliary differential filter is designed based on the adaptive kernel, and the initial feature map is binarized based on the auxiliary differential filter.

[0014] Furthermore, the step of using the center pixel as a local threshold for the neighboring pixels in the binarized feature map and collecting high-order image statistical information includes:

[0015] Local threshold calculation is performed by binarizing the neighboring pixels of the location in the initial feature map mapping;

[0016] The input channel in each convolutional layer is divided into two non-overlapping parts, which are used to capture zero-order or higher-order information respectively.

[0017] By adjusting the split ratio ξ, structures with different mixing schemes can be obtained.

[0018] Furthermore, the image processing that achieves local binary convolution by having a binary neural network recognize the higher-order image statistical information includes:

[0019] High-capacity information flow is facilitated by adding additional skip connections in each block.

[0020] The PReLU activation function is used to train the data;

[0021] In the residual path of each module, a preset sequence is used to obtain the activation distribution.

[0022] On the other hand, embodiments of the present invention also provide an image processing system based on local binary convolution, comprising:

[0023] The image acquisition module is used to acquire the original image to be processed, and to perform feature reprocessing on the original image to obtain an initial feature map.

[0024] The binarization feature module is used to construct a local binary convolution model based on a binary neural network, and to perform binarization processing on the initial feature map through the binary convolution model to obtain a binarized feature map.

[0025] The local threshold acquisition module is used to take the center pixel as the local threshold for the neighboring pixels in the binarized feature map and collect high-order image statistical information.

[0026] The image implementation module is used to perform image processing by enabling a binary neural network to recognize the higher-order image statistical information, thereby achieving local binary convolution.

[0027] Furthermore, the binarization feature module includes local convolutional units, which are used for:

[0028] An adaptive kernel is obtained by modifying the identity kernel in the original binary convolution to capture and retain the high-order information of the initial feature map during the preset binarization process;

[0029] An auxiliary differential filter is designed based on the adaptive kernel, and the initial feature map is binarized based on the auxiliary differential filter.

[0030] Furthermore, the local threshold acquisition module includes a gradient information unit, which is used for:

[0031] Local threshold calculation is performed by binarizing the neighboring pixels of the location in the initial feature map mapping;

[0032] The input channel in each convolutional layer is divided into two non-overlapping parts, which are used to capture zero-order or higher-order information respectively.

[0033] By adjusting the split ratio ξ, structures with different mixing schemes can be obtained.

[0034] Furthermore, the image implementation module includes a binary processing unit, which is used for:

[0035] High-capacity information flow is facilitated by adding additional skip connections in each block.

[0036] The PReLU activation function is used to train the data;

[0037] In the residual path of each module, a preset sequence is used to obtain the activation distribution.

[0038] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0039] Obtain the original image to be processed, and perform feature reprocessing on the original image to obtain an initial feature map;

[0040] A local binary convolution model is constructed based on a binary neural network. The initial feature map is then binarized using the binary convolution model to obtain a binarized feature map.

[0041] The neighboring pixels in the binarized feature map are used with the center pixel as the local threshold, and high-order image statistical information is collected.

[0042] Image processing with local binary convolution is achieved by having a binary neural network recognize the higher-order image statistical information.

[0043] This invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:

[0044] Obtain the original image to be processed, and perform feature reprocessing on the original image to obtain an initial feature map;

[0045] A local binary convolution model is constructed based on a binary neural network. The initial feature map is then binarized using the binary convolution model to obtain a binarized feature map.

[0046] The neighboring pixels in the binarized feature map are used with the center pixel as the local threshold, and high-order image statistical information is collected.

[0047] Image processing with local binary convolution is achieved by having a binary neural network recognize the higher-order image statistical information.

[0048] The aforementioned image processing method, system, device, and storage medium based on local binary convolution include: acquiring the original image to be processed; performing feature-based reprocessing on the original image to obtain an initial feature map; constructing a local binary convolution model based on a binary neural network; performing binarization processing on the initial feature map through the binary convolution model to obtain a binarized feature map; using the center pixel as a local threshold for neighboring pixels in the binarized feature map and collecting high-order image statistical information; and realizing local binary convolution image processing by having the binary neural network recognize the high-order image statistical information. This embodiment proposes a novel binary convolution operation method that can effectively capture high-order image statistical information from the input feature map during image processing. Furthermore, this method can effectively capture high-order image statistical information with maximized information retention from the input feature map through a local thresholding strategy, thereby improving the accuracy of image recognition processing. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating an image processing method based on local binary convolution in one embodiment.

[0050] Figure 2 This is a flowchart illustrating the design of a local binary convolution model in one embodiment.

[0051] Figure 3 This is a schematic diagram showing the comparison of LBConv and CBConv convolution kernels in one embodiment.

[0052] Figure 4 The frequency diagram of the LBConv and CBConv filter cores in one embodiment is shown.

[0053] Figure 5 This is a schematic diagram of the local threshold calculation process in one embodiment.

[0054] Figure 6 This is a flowchart illustrating the process of building a basic BNN module in one embodiment.

[0055] Figure 7 This is a schematic diagram of the basic modules in the BNN structure in one embodiment.

[0056] Figure 8 This is a block diagram of an image processing system based on local binary convolution in one embodiment.

[0057] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] In one embodiment, such as Figure 1 As shown, an image processing method based on local binary convolution is provided, the method comprising:

[0060] Step 101: Obtain the original image to be processed, and perform feature reprocessing on the original image to obtain an initial feature map;

[0061] Step 102: Construct a local binary convolution model based on the binary neural network, and perform binarization processing on the initial feature map through the binary convolution model to obtain a binarized feature map;

[0062] Step 103: Use the center pixel as the local threshold for the neighboring pixels in the binarized feature map, and collect high-order image statistical information.

[0063] Step 104: Image processing with local binary convolution is achieved by having the binary neural network recognize the higher-order image statistical information.

[0064] Specifically, binary neural networks (BNNs) are characterized by hardware friendliness, memory saving, and computational efficiency, especially when both weights and activations are limited to 1 bit. Recently, researchers have mitigated the accuracy degradation of BNNs by binarizing the full-precision network structure. In this invention, we propose a novel binary convolution operator, Local Binary Convolution (LBConv), to improve the accuracy of BNNs without sacrificing efficiency. Compared to traditional binary convolution (CBConv) which uses a single fixed threshold for activation binarization, the proposed LBConv uses the center pixel as the local threshold when binarizing neighboring pixels in the feature map. Therefore, unlike CBConv, LBConv can effectively collect rich high-order image statistical information, which we define as information reflecting intensity changes between local regions, such as gradient information. When considering both high-order and original zero-order information simultaneously, the discriminative power of the BNN structure can be significantly enhanced, resulting in higher prediction accuracy.

[0065] This embodiment proposes a novel binary convolution operation method. This method can effectively capture high-order image statistical information from the input feature map during image processing. Furthermore, this method can effectively capture high-order image statistical information with maximum information retention from the input feature map through a local thresholding strategy, thereby improving the accuracy of image recognition processing without increasing computational cost or parameter complexity.

[0066] In one embodiment, such as Figure 2 As shown, the process of designing a local binary convolution model includes the following steps:

[0067] Step 201: By modifying the identity kernel in the original binary convolution, an adaptive kernel is obtained to capture and retain the high-order information of the initial feature map during the preset binarization process.

[0068] Step 202: Design an auxiliary differential filter based on the adaptive kernel, and perform binarization processing on the initial feature map based on the auxiliary differential filter.

[0069] Specifically, pioneering BNN work suggested using full-precision latent weights during training to accumulate the gradients of the corresponding binary weights. With a few exceptions, latent weights have been widely used to update binary weights. Due to the non-differentiability of the Sign function, a pass-through estimator (STE) is also used to approximate the gradients of binarized activations and binary weights. Based on this, later research focused on improving gradient approximation during backpropagation, often through carefully designed functions instead of the original hard tanh function. Simultaneously, reducing the quantization error between full-precision and binary activations, or between full-precision latent weights and binary weights, is another research direction for improving BNNs. Solutions include introducing appropriate scaling factors or shift scalars, or using other strategies. On the other hand, due to the significant information loss after binarization, some current work uses additional skip connections to preserve activations before binarization, thus achieving a stronger information flow. Multiple binarization uses multiple bases or branches in the BNN to approximate the full-precision network. This also increases the network width to enhance representational power. Furthermore, some work uses Neural Architecture Search (NAS) techniques to search for accurate and efficient BNNs.

[0070] By incorporating discriminative high-order image statistics, this embodiment enhances the network's representational capabilities for better recognition. Furthermore, the proposed LBConv can achieve better information flow by increasing information entropy.

[0071] For example, suppose we have a feature map with a single input. Latent weighted convolution kernel and a single output feature map The convolution process is described. Here, h and w represent the height and width of the input / output feature maps, and k represents the kernel size. The binarized input feature map and binary weights are represented by X. b and W b This is indicated by... Furthermore, assuming the activation binarization threshold is 0, then Y implemented using a regular binary convolution (CBConv) is as follows:

[0072]

[0073] The above formula can also be expressed as:

[0074]

[0075] Where (u, v) represents the current pixel's position in X and Y, i.e., located in row u and column v, and N represents the position of the pixel in the x and y coordinates. The local neighborhood centered at X. It is worth noting that if (u+p,v+q) exceeds the image boundary of X, it is assigned a predefined padding value.

[0076] The above expression can be reformulated as:

[0077]

[0078] In the formula, v is a predefined fill value.

[0079] Then define For the auxiliary filter kernel set, in CBConv, all They all use the same convolutional kernel, that is ,like Figure 3 As shown in (a), only the intensity of a single pixel (zero-order information) is considered. Due to the extreme form of quantization under the sign function, it is difficult to preserve high-order image cues that represent local intensity changes between adjacent pixels.

[0080] In LBConv, following the paradigm of differential operators, the identity kernel in CBConv is modified. To capture and preserve higher-order information during the binarization process, the modified convolution kernel formula is as follows:

[0081]

[0082] like Figure 3 As shown in (b), LBConv uses an adaptive kernel modified from the unified convolution kernel, with hollow and solid circles representing -1 and +1 respectively, and does not use kernel values ​​without circles. For better implementation, it can be used for each Arrange filters during the convolution process. Figure 3 (b) All unfilled circles correspond to pixels at the same position in X.

[0083] For richness, the non-zero elements in the kernel can be distributed along different directions. For efficiency, we place only two non-zero elements (i.e., +1 and -1) in each kernel, so the convolution unit and the Sign function can be combined into a simple comparison operation, as follows:

[0084]

[0085] In the above formula, b can be seen as a threshold for binarizing a. Assuming m such auxiliary filters are used in the above convolution kernel formula, the +1 element can be fixed at the center of each filter kernel, and the -1 elements can be distributed in adjacent positions around the center. By... With the auxiliary differential filter arranged, the formula for Y implemented by local binary convolution (LBConv) is as follows:

[0086]

[0087] In the above formula This is the local threshold for binarizing the neighboring pixels at position (u,v) in the feature map X. The above formula also applies to larger convolutional kernel sizes. For example, when k=5, the center pixel can be used as the threshold to binarize the surrounding 24 pixels.

[0088] Furthermore, according to the convolution theorem and the modified convolution kernel formula described above, for 2D spatial signals, such as binarized feature maps, convolution in the spatial domain is equal to point-by-point multiplication in the frequency domain. Therefore, the convolution kernel formula can be further expressed as:

[0089]

[0090] FFT2 and IFFT2 represent the two-dimensional fast Fourier transform and the inverse two-dimensional fast Fourier transform, respectively. Through these transformations, the auxiliary differential filter can act as an attention map on the input feature map in the frequency domain, suppressing or highlighting specific frequency components.

[0091] like Figure 4 As shown, Figure 4 (a) is a frequency diagram of the identity filter core in CBConv and the auxiliary filter core in LBConv. Figure 4 (b) Frequency map of feature maps generated by CBConv (top row) and LBConv (bottom row) in convolutional layers of different depths.

[0092] To observe the effect of the auxiliary filter in the frequency domain, Figure 4 (a) illustrates the Fast Fourier Transform results of the identity kernel used in CBConv and the differential filter kernel used in LBConv, assuming the input feature map size is 32*32. It can be seen that LBConv actually employs a series of high-pass filters, focusing on extracting high-frequency image information from different spatial directions. On the other hand, a uniform convolution kernel does not highlight any frequency components of the input feature map. When the BNN only uses CBConv, the extracted features will be monopolized by inherent low-frequency components, drowning out high-frequency signals that might provide information and enhance the BNN's discriminative ability.

[0093] In this embodiment, Figure 4 (b) visualizes the actual feature maps in different layers generated by CBConv and LBConv in the frequency domain. By averaging along the batch and channel dimensions, the results of the corresponding fast Fourier transforms of 100 randomly sampled images from the ImageNet validation set can be shown that LBConv emphasizes more high-frequency components compared to CBConv.

[0094] Furthermore, for those with The entropy of the discrete variable x, representing a possible outcome, describes its information content as follows:

[0095] Higher entropy indicates that each bit of a variable contains more information. In the case of BNNs, the binary activation has only two possible values, and information is preserved to the maximum extent when these two values ​​are distributed with equal probability. However, in LBConv, maximizing the information entropy of the quantized values ​​is an inherent property.

[0096] Since the distribution of activation values ​​before binarization approximates a Gaussian distribution, the following assumption is made:

[0097] 1. A variable x representing the full-precision activation value at any position i. i It follows a Gaussian distribution.

[0098] 2. If i is not equal to j, x i and x j They are independent of each other.

[0099] 3. Variables in the same feature map share the same distribution parameters.

[0100] For features from the same feature map and following a Gaussian distribution x i and x j We can obtain:

[0101]

[0102] The formula for calculating probability P is as follows:

[0103]

[0104] Therefore, through the implementation formula of LBConv, the binarized activation value can achieve the maximum Shannon entropy, thereby obtaining the optimized information gain. Thus, LBConv also achieves the maximization of information entropy through local thresholding.

[0105] In one embodiment, such as Figure 5 As shown, the process for calculating the local threshold includes:

[0106] Step 301: Perform local threshold calculation by binarizing the neighboring pixels of the location in the initial feature map mapping;

[0107] Step 302: Divide the input channel in each convolutional layer into two non-overlapping parts, which are used to capture zero-order or higher-order information respectively;

[0108] Step 303: By adjusting the split ratio ξ, structures with different mixing schemes are obtained.

[0109] Specifically, CBConv and LBConv are essentially orthogonal operations that extract image information from different perspectives. Therefore, both can be considered as enhancing the representational power of the network. To this end, the input channels in each convolutional layer are divided into two non-overlapping parts, used to capture zero-order or higher-order information respectively. This is achieved by adjusting the split ratio. This allows for the acquisition of structures with different hybridization schemes.

[0110] Assuming it will have The input feature maps for each channel are fed into a binary convolutional layer, which should generate an output feature map with those channels. A binary convolution containing LBConv and CBConv can be written as:

[0111]

[0112] Where f L and f C It is a combination of other transformations, such as batch normalization (BN) and ReLU. Both the LBConv and CBConv branches will generate... The feature map of the channel, therefore, can be found in f L and f C Element-wise addition is then applied. The overall computational cost is not increased compared to a standard binary convolutional layer with pure CBConv.

[0113] The only difference between LBConv and CBConv proposed in this invention lies in the binarization method of the activations. Both involve binary matrix multiplication between binarized activation values ​​and binary weights. For example, daBNN uses "im2col" to expand the activations into a matrix, where each row represents an expanded local sampling region of the input feature map. In CBConv, each row's elements are binarized using a global threshold, while in LBConv, they are binarized using the center column pixel (corresponding to the center pixel of the local region) as the threshold. The resulting binarized activation matrix is ​​then multiplied by the binary weights, where the multiplication can be implemented using an efficient xnor-count operation.

[0114] In one embodiment, such as Figure 6 As shown, the process of building a basic BNN module includes:

[0115] Step 401, to facilitate high-capacity information flow by adding additional skip connections in each block;

[0116] Step 402: Use the PReLU activation function to train the data;

[0117] Step 403: In the residual path of each module, a preset sequence is used to obtain the activation distribution.

[0118] Specifically, the key aspect of this embodiment is to first construct a strong baseline model. First, additional skip connections are added to each block to facilitate high-capacity information flow. Second, the PReLU activation function is employed for better training. Finally, a "BN-BConv-BN" sequence is used in the residual path of each module to obtain a better activation distribution. For example... Figure 7 As shown, the baseline architecture corresponds to ξ=0 in the basic block. A BNN based on LBConv (LBNN) can be constructed by setting ξ>0. ξ is a parameter that controls the influence of LBConv and CBConv. This indicates element-wise addition.

[0119] Based on the above process steps, this embodiment uses an image classification experiment for verification. Unlike complex multi-step training schemes, the verification process uses the Adam optimizer to train all models from scratch with a moderate number of training iterations. In the BNN model, the first convolutional layer, the convolutional layers in the skip connections, and the last fully connected layer are retained at full precision. Gradient backpropagation is performed using STE in the same manner.

[0120] In the image classification experiments of this embodiment, Cifar-10 and Cifar-100 each used 5000 images for training and 1000 images for testing, with 10 and 100 classes respectively. During training, images were augmented using random padding and cropping to a 32*32 size and random horizontal flipping. The experiments were based on the ResNet-18 architecture. The initial learning rate was set to 0.001 and reduced to 1 / 10 at epochs 45 and 55, with a batch size of 64. To obtain a reliable experimental record on the Cifar dataset, each setup was run five times independently, and the mean and standard deviation were calculated.

[0121] To observe how the proposed LBConv affects the classification prediction performance of the BNN, different values ​​of ξ were set on the LBNN structure. For both datasets, compared with the baseline model, the accuracy began to improve by gradually increasing the value of ξ. This is because more higher-order gradient information is introduced by incorporating LBConv. After peaking at ξ=0.2, the accuracy began to decline. There is an extreme case at ξ=1.0, where the model only uses LBConv, and the performance is worse than the baseline model, indicating that to maximize the performance of the BNN, it is necessary to consider the discriminative power of both zero-order and higher-order information.

[0122] The ImageNet (ILSVRC2012) dataset contains 1.2 million images for training and 50,000 images for validation, with 1000 classes. During training, images are augmented through random cropping and random horizontal flipping. The learning rate is set the same as the experimental settings on the Cifar dataset, with a batch size of 256. Based on observations of the Cifar dataset, ξ=0.2 is set in the LBNN. Due to the improved layer order (i.e., the first BN layer approximates RSign with a learnable bias / threshold, and the second BN layer approximates RPReLU with a learnable scaling factor and bias), the baseline architecture achieves performance comparable to ReActNet.

[0123] By introducing higher-order information into LBNN, the performance of both architectures was significantly improved, with Top-1 accuracy increasing from 60.3% to 61.3% on ResNet-18 and from 64.6% to 65.4% on ResNet-34, further demonstrating the effectiveness of the proposed LBConv.

[0124] The experimental results are shown in Table 1:

[0125] Table 1

[0126]

[0127] In Table 1, W / A represents the number of bits used for weights and activations, respectively, and the last two rows are based on the baseline and LBNN models, respectively.

[0128] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0129] In another embodiment of the invention, a face recognition scenario can be constructed to conduct face recognition experiments. Specifically, the CASIA-WebFace dataset is used for training, and the face images are collected from the internet. After removing mislabeled images, 450,000 images usable for training are retained. Each image is aligned with the five facial landmarks detected by MTCNN and cropped to a size of 112*96. During testing, the face representation is obtained by concatenating features from the original image and the horizontally flipped image.

[0130] A BNN model was built based on a 20-layer CNN architecture, using A-SoftMax loss with an angular margin of m=4. The initial learning rate was 0.001, decaying to 1 / 10 of the original rate at epochs 12 and 17. The batch size was 128. Similarly, ξ=0.2 was set in the LBNN. The test results are shown in Table 2. In all four test datasets, the BNN using LBConv consistently outperformed the baseline structure.

[0131] Table 2

[0132]

[0133] In one embodiment, such as Figure 8 As shown, an image processing system based on local binary convolution is provided, including:

[0134] The image acquisition module 501 is used to acquire the original image to be processed, and to perform feature reprocessing on the original image to obtain an initial feature map.

[0135] The binarization feature module 502 is used to construct a local binary convolution model based on a binary neural network, and to perform binarization processing on the initial feature map through the binary convolution model to obtain a binarized feature map.

[0136] The local threshold acquisition module 503 is used to take the center pixel as the local threshold for the neighboring pixels in the binarized feature map and collect high-order image statistical information.

[0137] The image implementation module 504 is used to perform image processing by enabling a binary neural network to recognize the higher-order image statistical information, thereby achieving local binary convolution.

[0138] In one embodiment, such as Figure 8 As shown, the binarization feature module 502 includes a local convolution unit 5021, which is used for:

[0139] An adaptive kernel is obtained by modifying the identity kernel in the original binary convolution to capture and retain the high-order information of the initial feature map during the preset binarization process;

[0140] An auxiliary differential filter is designed based on the adaptive kernel, and the initial feature map is binarized based on the auxiliary differential filter.

[0141] In one embodiment, such as Figure 8 As shown, the local threshold acquisition module 503 includes a gradient information unit 5031, which is used for:

[0142] Local threshold calculation is performed by binarizing the neighboring pixels of the location in the initial feature map mapping;

[0143] The input channel in each convolutional layer is divided into two non-overlapping parts, which are used to capture zero-order or higher-order information respectively.

[0144] By adjusting the split ratio ξ, structures with different mixing schemes can be obtained.

[0145] In one embodiment, such as Figure 8 As shown, the image implementation module 504 includes a binary processing unit 5041, which is used for:

[0146] High-capacity information flow is facilitated by adding additional skip connections in each block.

[0147] The PReLU activation function is used to train the data;

[0148] In the residual path of each module, a preset sequence is used to obtain the activation distribution.

[0149] Specific limitations regarding the image processing system based on local binary convolution can be found in the limitations of the image processing method based on local binary convolution mentioned above, and will not be repeated here. Each module in the aforementioned image processing system based on local binary convolution can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0150] Figure 9 An internal structural diagram of a computer device in one embodiment is shown. Figure 9As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement an image processing method based on local binary convolution. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the image processing method based on local binary convolution. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0151] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0153] Obtain the original image to be processed, and perform feature reprocessing on the original image to obtain an initial feature map;

[0154] A local binary convolution model is constructed based on a binary neural network. The initial feature map is then binarized using the binary convolution model to obtain a binarized feature map.

[0155] The neighboring pixels in the binarized feature map are used with the center pixel as the local threshold, and high-order image statistical information is collected.

[0156] Image processing with local binary convolution is achieved by having a binary neural network recognize the higher-order image statistical information.

[0157] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0158] By modifying the identity kernel in the original binary convolution, the ability to capture and retain high-order information in the initial feature map during the binarization process can be preset.

[0159] An auxiliary differential filter is designed based on the modified identity kernel, and the initial feature map is binarized based on the auxiliary differential filter.

[0160] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0161] The step of using the center pixel as a local threshold for neighboring pixels in the binarized feature map and collecting high-order image statistical information includes:

[0162] Local threshold calculation is performed by binarizing the neighboring pixels of the location in the initial feature map mapping;

[0163] The input channel in each convolutional layer is divided into two non-overlapping parts, which are used to capture zero-order or higher-order information respectively.

[0164] By adjusting the split ratio ξ, structures with different mixing schemes can be obtained.

[0165] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0166] High-capacity information flow is facilitated by adding additional skip connections in each block.

[0167] The PReLU activation function is used to train the data;

[0168] In the residual path of each module, a preset sequence is used to obtain the activation distribution.

[0169] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0170] Obtain the original image to be processed, and perform feature reprocessing on the original image to obtain an initial feature map;

[0171] A local binary convolution model is constructed based on a binary neural network. The initial feature map is then binarized using the binary convolution model to obtain a binarized feature map.

[0172] The neighboring pixels in the binarized feature map are used with the center pixel as the local threshold, and high-order image statistical information is collected.

[0173] Image processing with local binary convolution is achieved by having a binary neural network recognize the higher-order image statistical information.

[0174] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0175] An adaptive kernel is obtained by modifying the identity kernel in the original binary convolution to capture and retain the high-order information of the initial feature map during the preset binarization process;

[0176] An auxiliary differential filter is designed based on the adaptive kernel, and the initial feature map is binarized based on the auxiliary differential filter.

[0177] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0178] The step of using the center pixel as a local threshold for neighboring pixels in the binarized feature map and collecting high-order image statistical information includes:

[0179] Local threshold calculation is performed by binarizing the neighboring pixels of the location in the initial feature map mapping;

[0180] The input channel in each convolutional layer is divided into two non-overlapping parts, which are used to capture zero-order or higher-order information respectively.

[0181] By adjusting the split ratio ξ, structures with different mixing schemes can be obtained.

[0182] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0183] High-capacity information flow is facilitated by adding additional skip connections in each block.

[0184] The PReLU activation function is used to train the data;

[0185] In the residual path of each module, a preset sequence is used to obtain the activation distribution.

[0186] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method of image processing based on local binary convolution, characterized in that, The method comprises: acquiring an original image to be processed, performing reprocessing of featureization on the original image to obtain an initial feature map; constructing a local binary convolution model according to a binary neural network, performing binaryzation processing on the initial feature map through the binary convolution model, and obtaining a binaryzation feature map; using a center pixel as a local threshold for neighborhood pixels in the binaryzation feature map, and collecting high-order image statistical information, including: local threshold calculation of binaryzation of neighborhood pixels in the initial feature map mapping position; dividing the input channel in each convolution layer into two non-overlapping parts, respectively used for capturing zero-order or high-order information; obtaining a structure with different mixing schemes by adjusting the shunt ratio ξ; realizing image processing of local binary convolution by letting the binary neural network recognize the high-order image statistical information.

2. The local binary convolution-based image processing method of claim 1, wherein, The method comprises: obtaining an adaptive kernel by modifying an identity kernel in the original binary convolution, so as to capture and retain high-order information of the initial feature map in the preset binaryzation process; designing an auxiliary differential filter according to the adaptive kernel, and performing binaryzation processing on the initial feature map according to the auxiliary differential filter.

3. The local binary convolution-based image processing method of claim 1, wherein, The method comprises: adding an additional jump connection in each block to promote high-capacity information flow; using a PReLU activation function to realize training of data; using a preset sequence in the residual path of each module to obtain an activation distribution.

4. A local binary convolution based image processing system, characterized by, The method comprises: an image acquisition module, configured to acquire an original image to be processed, perform reprocessing of featureization on the original image, and obtain an initial feature map; a binaryzation feature module, configured to construct a local binary convolution model according to a binary neural network, perform binaryzation processing on the initial feature map through the binary convolution model, and obtain a binaryzation feature map; a local threshold acquisition module, configured to use a center pixel as a local threshold for neighborhood pixels in the binaryzation feature map, and collect high-order image statistical information; the local threshold acquisition module comprises a gradient information unit, which is configured to: perform local threshold calculation of binaryzation of neighborhood pixels in the initial feature map mapping position; divide the input channel in each convolution layer into two non-overlapping parts, respectively used for capturing zero-order or high-order information; and obtain a structure with different mixing schemes by adjusting the shunt ratio ξ; an image realization module, configured to realize image processing of local binary convolution by letting the binary neural network recognize the high-order image statistical information.

5. The local binary convolution-based image processing system of claim 4, wherein, The binaryzation feature module comprises a local convolution unit, which is configured to: obtain an adaptive kernel by modifying an identity kernel in the original binary convolution, so as to capture and retain high-order information of the initial feature map in the preset binaryzation process; design an auxiliary differential filter according to the adaptive kernel, and perform binaryzation processing on the initial feature map according to the auxiliary differential filter.

6. The local binary convolution-based image processing system of claim 4, wherein, The image realization module comprises a binary processing unit, which is used for: promoting high-capacity information flow by adding additional skip connections in each block; training data by using a PReLU activation function; in the residual path of each module, a preset sequence is used to obtain an activation distribution.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1 to 3.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 3.

Images