Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a to-be-processed input image; sending the input image into a feature extraction module to extract features from the input image layer by layer to generate a feature map; sending the feature maps into a feature fusion module, and performing up-sampling and splicing operation on the feature maps to fuse the feature maps from different levels to obtain a fused feature map; sending the fused feature map into an output module, performing up-sampling operation on the fused feature map to adjust the spatial resolution of the fused feature map, so that the fused feature map is matched with the spatial resolution of the input image, and generating a matched feature map; and performing convolution operation on the matched feature map to generate a single-channel feature map. According to the invention, the neural network structure is optimized, so that the problem of dependence on high-performance hardware is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a neural network-based image processing method, device, electronic device and storage medium. Background Art

[0002] In the current industry, there are three main methods for identifying and locating text positions on images: morphological operation methods, object detection methods, and segmentation network methods. Morphological operation methods use basic morphological transformations to identify text areas, but their positioning accuracy is low. Object detection methods, such as CTPN (Connectionist Text Proposal Network), YOLO (You Only Look Once), and SSD (Single Shot MultiBox Detector), provide relatively accurate positioning, but still have the problem of insufficient accuracy and increase the difficulty of subsequent text recognition. Segmentation network methods, such as EAST (Efficient and Accurate Scene Text Detector) and DB (Differentiable Binarization), can achieve accurate text position positioning, but due to their large network structure and numerous algorithm model weight parameters, they require high-performance hardware support, so there are certain limitations in practicality. Summary of the invention

[0003] The present application provides an image processing method, device, electronic device and storage medium to solve the problem of how to optimize the neural network structure to reduce the dependence on high-performance hardware.

[0004] In a first aspect, the present application provides an image processing method, which is applied to an image processing device, wherein the image processing device includes a neural network, wherein the neural network includes a feature extraction module, a feature fusion module, and an output module, and the image processing method includes:

[0005] Get the input image to be processed;

[0006] Sending the input image to the feature extraction module to extract features from the input image layer by layer to generate a feature map;

[0007] The feature map is sent to the feature fusion module, and the feature maps from different levels are fused by performing upsampling and concatenation operations on the feature map to obtain a fused feature map;

[0008] Sending the fused feature map to the output module, performing an upsampling operation on the fused feature map to adjust its spatial resolution so that it matches the spatial resolution of the input image, and generating a matching feature map; and

[0009] A convolution operation is performed on the matching feature map to generate a single-channel feature map, where the single-channel feature map is used to identify whether a text area exists in the input image.

[0010] In an embodiment of the present application, after acquiring the input image to be processed, the image processing method further includes:

[0011] The input image is preprocessed to adjust the input image to a preset size and a preset number of channels.

[0012] In one embodiment of the present application, the feature extraction module includes multiple convolutional layers and multiple pooling layers; wherein the multiple convolutional layers are used to gradually increase the number of channels of the feature map and keep the spatial resolution of the feature map constant, and the multiple pooling layers are used to reduce the spatial resolution of the feature map layer by layer.

[0013] In an embodiment of the present application, the step of sending the feature map to the feature fusion module, performing upsampling and splicing operations on the feature map to fuse feature maps from different levels, and obtaining a fused feature map includes:

[0014] Performing an upsampling operation on the feature map extracted by the final level of the feature extraction module;

[0015] Concatenate the upsampled feature map with the feature map extracted from the next previous level;

[0016] The concatenated feature map is then upsampled and concatenated with the feature map extracted from the previous level.

[0017] This process is repeated until the feature map extracted at the initial level is processed, and finally the fused feature map that fuses feature information of all levels is obtained.

[0018] In one embodiment of the present application, the neural network structure includes multiple levels, each level is composed of several convolutional layers and at least one pooling layer; the convolutional layers and pooling layers in each level work together to extract features at different levels of the input image; the levels are arranged from front to back, the initial level is responsible for extracting low-level features, and the subsequent levels are responsible for extracting higher-level features; the final level is responsible for outputting the results after network processing, and the initial level receives the input image.

[0019] In one embodiment of the present application, in the neural network, the input of each layer first passes through a convolutional layer, the output feature map of the convolutional layer is used as the input of the next convolutional layer, the output of the second convolutional layer is used as the input of the pooling layer, and the output of the pooling layer is passed to the first convolutional layer of the next layer for step-by-step feature extraction, and each layer further extracts features based on the previous layer.

[0020] In one embodiment of the present application, in the neural network, the output after processing by each convolutional layer is connected to a preset activation function to perform nonlinear transformation on the feature map output by the convolutional layer.

[0021] In one embodiment of the present application, after performing the step of performing a convolution operation on the matching feature map to generate a single-channel feature map, the image processing method further includes:

[0022] Analyzing each feature point in the single-channel feature map to determine whether they indicate that the corresponding pixel point in the input image belongs to a text area;

[0023] For each feature point, if its value is higher than a preset threshold, it is determined that the pixel point in the input image corresponding to the feature point belongs to the text area;

[0024] The determination results of all feature points are integrated, and the position information of the text area of ​​the input image is identified by counting the positions of the feature points above the threshold in the single-channel feature map.

[0025] In a second aspect, the present application further provides an image processing device, the image processing device comprising:

[0026] A receiving module, used for obtaining an input image to be processed;

[0027] A neural network, comprising a feature extraction module, a feature fusion module and an output module; the feature extraction module is responsible for extracting features from the input image layer by layer and generating a feature map; the feature fusion module obtains a fused feature map by performing upsampling and splicing operations on the feature map to fuse feature maps from different layers; the output module first upsamples the fused feature map to adjust its spatial resolution to match the input image, thereby generating a matching feature map; finally, the output module performs a convolution operation on the matching feature map to generate a single-channel feature map, which is used to identify whether there is a text area in the input image.

[0028] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the image processing method described in the first aspect are implemented.

[0029] In a fourth aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the image processing method as described in any one of the first aspects are implemented.

[0030] The image processing method, device, electronic device and storage medium provided by the present application reduce the dependence on high-performance hardware by optimizing the neural network structure, which is mainly reflected in: First, by extracting features from the input image layer by layer, the network can capture the features of the image at different scales. This is more effective than single-scale feature extraction because it can adapt to text areas of different sizes, thereby reducing the demand for high-performance hardware. Secondly, the upsampling operation can restore the spatial resolution of the feature map, and the splicing operation fuses the feature maps of different levels, which can make full use of multi-scale information and improve the accuracy of detection. Finally, the output module adjusts the spatial resolution of the fused feature map through the upsampling operation to match the input image, and generates a single-channel feature map through the convolution operation. Single-channel feature maps mean less computation and memory requirements, which helps to run the neural network on lower-performance hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 is a flowchart of an image processing method provided in an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0034] Figure 3 It is one of the schematic diagrams of the activation function provided in the embodiment of the present application;

[0035] Figure 4 This is the second schematic diagram of the activation function provided in the embodiment of the present application;

[0036] Figure 5 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0037] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0040] In order to solve the problem of how to optimize the neural network structure to reduce the dependence on high-performance hardware, the present application provides an image processing method, device, electronic device and storage medium. The image processing method reduces the number and dimension of feature maps by introducing the steps of extracting image features layer by layer, fusion of feature maps and generation of single-channel feature maps in the neural network, thereby reducing the dependence of the entire neural network on high-performance hardware. Such optimization enables the neural network to run on devices with limited computing resources, while also improving processing speed and efficiency.

[0041] Combine the following Figure 1-Figure 6 The present invention describes an image processing method, an apparatus, an electronic device and a storage medium.

[0042] Please refer to Figure 1 , Figure 1 1 is a flow chart of an image processing method provided by an embodiment of the present application. An image processing method is applied to an image processing device, wherein the image processing device includes a neural network, the neural network includes a feature extraction module, a feature fusion module and an output module, and the image processing method includes:

[0043] S101, obtaining an input image to be processed.

[0044] S102, sending the input image to a feature extraction module to extract features from the input image layer by layer to generate a feature map.

[0045] Exemplarily, the feature extraction module includes multiple convolutional layers and multiple pooling layers. Specifically, the role of multiple convolutional layers is to gradually increase the number of channels of the feature map, that is, to increase the diversity of features, while keeping the spatial resolution of the feature map unchanged, which means that the size of the feature map remains constant after being processed by the convolutional layer. The role of multiple pooling layers is to reduce the spatial resolution of the feature map at each level, that is, to reduce the size of the feature map, which can reduce the amount of data for subsequent processing while retaining important feature information. Through such a structural design, the feature extraction module can effectively extract useful feature information from the input data.

[0046] Exemplarily, a neural network includes multiple layers. Each layer contains several convolutional layers and one or more pooling layers. These layers are arranged in order from front to back, forming a hierarchical structure for feature extraction. Within each layer, the convolutional layer and the pooling layer work together to extract features at different levels of the input image. The first layer is mainly responsible for extracting basic, low-level features of the image, such as edges and corners. As the layers move backward, the subsequent layers gradually extract more complex and advanced features, which are more abstract and reflect the deep structure and semantic information of the image. The final layer is responsible for outputting the results of network processing. The initial layer directly receives the input image as the starting point of network processing. Through such a hierarchical structure, the neural network can effectively extract and combine features from the input data, and ultimately achieve high-level understanding and processing of the image.

[0047] Exemplarily, the feature extraction process of each layer in the neural network includes: first, the input data of each layer passes through a convolution layer. The convolution layer uses a set of learnable filters (or convolution kernels) to scan the input data, extract local features, and generate feature maps. The output of the convolution layer (i.e., the feature map) is then used as the input of the next convolution layer. The second convolution layer applies a set of filters again to further extract features and generate a new feature map. The output of the second convolution layer is then used as the input of the pooling layer. The pooling layer reduces the spatial resolution of the feature map through downsampling operations while retaining important feature information, which helps to reduce the amount of computation and enhance the robustness of the features. The output of the pooling layer is passed to the first convolution layer of the next layer. In this way, each layer further extracts deeper features based on the features extracted by the previous layer.

[0048] Through this layer-by-layer processing mechanism, the neural network can gradually extract and combine features in the input data, thereby achieving high-level understanding and processing of the data. The convolutional layer and pooling layer at each level work together to ensure the gradual improvement of the hierarchy and abstractness of the features.

[0049] Exemplarily, in a neural network, after each convolutional layer processes the input data and generates a feature map, its output passes through a pre-set activation function. The function of the activation function is to perform a nonlinear transformation on the feature map output by the convolutional layer. Through this nonlinear transformation, the activation function can introduce the nonlinear characteristics of the model, allowing the network to learn and represent more complex data relationships and patterns. By applying the activation function after the convolutional layer, the expressive power of the neural network is enhanced, so that it can better process and classify complex data.

[0050] S103, sending the feature map to the feature fusion module, performing upsampling and concatenation operations on the feature map to fuse feature maps from different levels, and obtaining a fused feature map.

[0051] Exemplarily, the feature fusion process of the feature extraction module includes: performing an upsampling operation on the feature map extracted by the final level (i.e., the deepest level) of the feature extraction module. Upsampling is an operation that increases the spatial resolution of the feature map, which can enlarge the size of the feature map by interpolation and other methods. The upsampled feature map is spliced ​​with the feature map extracted by the immediately previous level (i.e., the second to last level). The splicing operation generally refers to connecting two feature maps in the channel dimension to form a feature map with more channels. The upsampling operation is performed again on the spliced ​​feature map to increase its spatial resolution. The upsampled feature map is spliced ​​with the feature map extracted by the next higher level (i.e., the third to last level). Repeat this process, splicing the upsampled feature map with the feature map of the next higher level each time, until the feature map extracted by the initial level (i.e., the shallowest level) is processed.

[0052] Finally, through this series of repeated operations, a fused feature map that integrates all levels of feature information is obtained. This fused feature map contains all features from low-level to high-level. This process allows the neural network to understand and process images at different spatial scales, thereby improving the ability to detect objects of different sizes or the ability to understand the content of the image.

[0053] S104 sends the fused feature map to the output module, performs an upsampling operation on the fused feature map to adjust its spatial resolution to match the spatial resolution of the input image, and generates a matching feature map.

[0054] S105, performing a convolution operation on the matching feature map to generate a single-channel feature map, where the single-channel feature map is used to identify whether there is a text area in the input image.

[0055] Exemplarily, the text area detection process based on a single-channel feature map includes: analyzing each feature point in the single-channel feature map. For each feature point, check whether its value is higher than a preset threshold. This threshold is a predetermined value used to distinguish whether the feature point indicates a text area. If the value of a feature point is higher than the threshold, it is determined that the pixel point in the input image corresponding to the feature point belongs to the text area. This means that the pixel point may be located in the text part of the image. The determination results of all feature points are integrated. This means that the determination results of all feature points in the feature map are summarized. By counting the positions of feature points above the threshold in the single-channel feature map, the position information of the text area in the input image can be identified. This position information can help determine which areas in the image contain text.

[0056] Therefore, in the above S101 to S105, first in step S101, the system obtains an input image to be processed as the original input image. Then, in step S102, the input image is sent to the feature extraction module of the neural network, which uses the convolution layer to extract key features from the image layer by layer and generates feature maps of each layer, which capture the feature information in the input image. Then, in step S103, the extracted feature map is passed to the feature fusion module, where the feature maps from different layers are fused together through upsampling and splicing operations to form a fused feature map containing multi-level information. In step S104, the fused feature map is sent to the output module, which performs an upsampling operation on the feature map to adjust its spatial resolution to match the resolution of the original input image, thereby generating a matching feature map. This step ensures that subsequent processing can be performed at the same spatial resolution as the original image, which is conducive to maintaining image details and accuracy. Finally, in step S105, a convolution operation is performed on the matching feature map to extract a single-channel feature map. A single-channel feature map refers to a feature map with only one channel, which focuses on identifying whether there is a text area in the image.

[0057] The above steps S101 to S105 are described in detail below through examples.

[0058] For example, please refer to Figure 2 , Figure 2 Schematic diagram of the structure of a neural network provided in an embodiment of the present application. An image processing method includes:

[0059] S201, obtaining an input image and preprocessing it.

[0060] Exemplarily, the input image is adjusted to a preset size and a preset number of channels. For example, the input data is preprocessed into an image with a size of 480x480 pixels and three color channels of R, G, and B, and the pixel values ​​of the image are normalized so as to be used as the input of the neural network. Normalization helps to improve the reasoning efficiency and stability of the neural network to prevent the scale differences between different features from being too large, thereby affecting the learning process of the network.

[0061] S202, input the preprocessed result to the first convolutional layer of the feature extraction module of the neural network to perform feature extraction.

[0062] For example, in Figure 2 In the example shown, the feature extraction module of the neural network consists of five layers, each of which consists of two convolutional layers and one pooling layer, as follows:

[0063] The first level: includes the first convolutional layer, the second convolutional layer and the first pooling layer.

[0064] The second level: includes the third convolution layer, the fourth convolution layer and the second pooling layer.

[0065] The third level: includes the fifth convolutional layer, the sixth convolutional layer and the third pooling layer.

[0066] The fourth level: includes the seventh convolution layer, the eighth convolution layer and the fourth pooling layer.

[0067] The fifth level: includes the ninth convolution layer, the tenth convolution layer and the fifth pooling layer.

[0068] For example, the output of S201 is used as the input of S202. The first convolution layer uses a 3x3x3 convolution kernel for convolution, with a total of 32 convolution kernels, a step size of 1, and an output feature map size of 480x480x32.

[0069] That is, the size of the convolution kernel (also called filter or weight matrix) is 3x3x3. The 3x3 here refers to the size of the convolution kernel in the width and height direction of the image, and the last 3 refers to the number of channels of the input feature map (3 in this case, corresponding to the three color channels of the RGB image). The convolution kernel slides on the input feature map, and each time a weighted sum is performed on the covered local area to extract features. There are 32 such convolution kernels. Each convolution kernel generates an output channel, so 32 convolution kernels will produce 32 different output channels, each channel representing a different feature extracted from the input. A stride of 1 means that the convolution kernel moves one pixel each time, which ensures that the convolution kernel covers all positions of the input feature map. After the operation of the first convolution layer, the resulting feature map size is 480x480x32.

[0070] In the entire neural network, the output of each convolutional layer is connected to a preset activation function to enhance the expression of nonlinear features. That is, the outputs of the first to tenth convolutional layers are all connected to a preset activation function.

[0071] For example, the preset activation function uses the hardswish activation function, which is written as a piecewise function:

[0072]

[0073] Among them, β is a trainable parameter or a custom parameter. When the value of β tends to infinity, the properties of the HardSwish activation function will approach a step function, such as Figure 3 As shown. A step function is a nonlinear function whose output jumps from 0 to 1 at a certain threshold. In this case, the output of the HardSwish activation function will become very close to a step function, that is, for inputs greater than a certain threshold, the output is close to 1, and for inputs less than the threshold, the output is close to 0.

[0074] The HardSwish activation function in the neural network of this application can convert linear output into nonlinear output, which helps the model capture the nonlinear relationship in the input data. The image of the derivative of the HardSwish activation function to x is as follows Figure 4 As shown in the figure. The HardSwish activation function improves the stability and efficiency of chain derivation in deep learning through its smoother derivatives, which helps the model update weights more accurately and improves training performance and speed. Compared with traditional ReLU and Leaky ReLU activation functions, HardSwish can better solve the problem of gradient disappearance or explosion when training deep networks, thereby improving the training effect of the model. Therefore, the HardSwish activation function is connected to the output of each convolutional layer, which can enhance the nonlinear ability of the model and improve the learning ability of the model while maintaining computational efficiency, making the model more effective in processing complex tasks.

[0075] S203: Input the result of the first convolutional layer to the second convolutional layer for feature extraction.

[0076] For example, the output of S202 is used as the input of S203. The second convolution layer uses a 1x1x32 convolution kernel for convolution, with a total of 32 convolution kernels and a step size of 1. The size of the resulting feature map is 480x480x32. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0077] S204, input the result of the second convolutional layer to the first pooling layer for feature extraction.

[0078] For example, the output of S203 is used as the input of S204. The first pooling layer uses a 6x6 pooling kernel for maximum pooling, with a total of 32 pooling kernels, a step size of 2, and an output feature map size of 240x240x32. Compared with the upper layer output, the feature map spatial resolution is reduced.

[0079] In digital images, spatial resolution is directly related to the pixel dimensions of the image. For example, a 480x480 pixel image has a higher spatial resolution than a 240x240 pixel image because it contains more pixels and is able to capture more detail.

[0080] S205, input the result of the first pooling layer to the third convolutional layer for feature extraction.

[0081] For example, the output of S204 is used as the input of S205. The third convolution layer uses a 3x3x32 convolution kernel for convolution, with a total of 64 convolution kernels and a step size of 1. The size of the resulting feature map is 240x240x64. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0082] S206: Input the result of the third convolutional layer to the fourth convolutional layer for feature extraction.

[0083] For example, the output of S205 is used as the input of S206. The fourth convolution layer uses a 1x1x64 convolution kernel for convolution, with a total of 64 convolution kernels and a step size of 1. The size of the resulting feature map is 240x240x64. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0084] S207, input the result of the fourth convolutional layer to the second pooling layer for feature extraction.

[0085] For example, the output of S206 is used as the input of S207. The second pooling layer uses a 5x5 pooling kernel for maximum pooling, with a total of 64 pooling kernels and a step size of 2. The size of the resulting feature map is 120x120x64. Compared with the upper layer output, the spatial resolution of the feature map is reduced.

[0086] S208, input the result of the second pooling layer to the fifth convolutional layer for feature extraction.

[0087] For example, the output of S207 is used as the input of S208. The fifth convolution layer uses a 3x3x64 convolution kernel for convolution, with a total of 128 convolution kernels and a step size of 1. The size of the resulting feature map is 120x120x128. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0088] S209: Input the result of the fifth convolutional layer to the sixth convolutional layer for feature extraction.

[0089] For example, the output of S208 is used as the input of S209. The sixth convolution layer uses a 1x1x128 convolution kernel for convolution, with a total of 128 convolution kernels and a step size of 1. The size of the resulting feature map is 120x120x128. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0090] S210, input the result of the sixth convolutional layer to the third pooling layer for feature extraction.

[0091] For example, the output of S209 is used as the input of S210. The third pooling layer uses a 4x4 pooling kernel for maximum pooling, with a total of 128 pooling kernels and a step size of 2. The size of the resulting feature map is 60x60x128. Compared with the upper layer output, the spatial resolution of the feature map is reduced.

[0092] S211, input the result of the third pooling layer to the seventh convolutional layer for feature extraction.

[0093] For example, the output of S210 is used as the input of S211. The seventh convolution layer uses a 3x3x128 convolution kernel for convolution, with a total of 256 convolution kernels and a step size of 1. The size of the resulting feature map is 60x60x256. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0094] S212, input the result of the seventh convolutional layer to the eighth convolutional layer for feature extraction.

[0095] For example, the output of S211 is used as the input of S212. The seventh convolution layer uses a 1x1x256 convolution kernel for convolution, with a total of 256 convolution kernels and a step size of 1. The size of the resulting feature map is 60x60x256. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0096] S213, input the result of the eighth convolutional layer to the fourth pooling layer for feature extraction.

[0097] For example, the output of S212 is used as the input of S213, and the fourth pooling layer uses a 3x3 pooling kernel for maximum pooling, with a total of 256 pooling kernels and a step size of 2. The size of the resulting feature map is 30x30x256. Compared with the upper layer output, the spatial resolution of the feature map is reduced.

[0098] S214, input the result of the fourth pooling layer to the ninth convolutional layer for feature extraction.

[0099] For example, the output of S213 is used as the input of S214, and the ninth convolution layer uses a 3x3x256 convolution kernel for convolution, with a total of 256 convolution kernels and a step size of 1. The size of the resulting feature map is 30x30x256. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0100] S215, input the result of the ninth convolutional layer to the tenth convolutional layer for feature extraction.

[0101] For example, the output of S214 is used as the input of S215, and the tenth convolution layer uses a 1x1x256 convolution kernel for convolution, with a total of 256 convolution kernels and a step size of 1. The size of the resulting feature map is 30x30x256. Compared with the upper layer output, the spatial resolution of the feature map is not changed.

[0102] S216, input the result of the tenth convolutional layer to the fifth pooling layer for feature extraction.

[0103] For example, the output of S215 is used as the input of S216, and the fifth pooling layer uses a 2x2 pooling kernel for maximum pooling, with a total of 256 pooling kernels and a step size of 2. The size of the resulting feature map is 15x15x256. Compared with the upper layer output, the spatial resolution of the feature map is reduced.

[0104] The above steps S202 to S216 describe the feature extraction process of the feature extraction module of the neural network for the input image, including convolution operations and pooling operations, which gradually extract the features of the input image, reduce the spatial size of the feature map, and increase the depth (i.e., the number of channels) of the feature map. The size of the feature map output by each layer depends on the size of the feature map of the previous layer, as well as the convolution kernel size, pooling kernel size, and step size. In this way, the feature extraction module can gradually extract more advanced and abstract features from the original input image. Finally, a feature map with low spatial resolution but strong feature expression ability is obtained.

[0105] Furthermore, in the above steps S202 to S216, the number of convolution kernels of each convolution layer is 32, 32, 64, 64, 128, 128, 256, 256, 256, 256. Due to the small number of convolution kernels, the weight parameters of the model will be reduced, which helps to reduce the complexity and memory requirements of the model. Moreover, the reduction of model weight parameters usually makes the model more efficient during reasoning because the amount of data and calculation required to be processed are reduced. Secondly, the size of the pooling kernel of each pooling layer is 6*6, 5*5, 4*4, 3*3, 2*2. By using pooling kernels of different sizes, it is more flexible when extracting features, and the size of the receptive field can be better balanced to avoid over-extraction of the receptive field around a single pixel, thereby possibly improving the robustness of the model. Moreover, the HardSwish activation function is introduced at the output of each convolution layer. The introduction of the HardSwish activation function helps the model converge faster because it can reduce the problem of gradient disappearance while maintaining nonlinear characteristics. Therefore, when processing text segmentation in small object scenes, this application will provide faster reasoning speed and better model robustness.

[0106] S217, performing an upsampling operation on the processing result of the fifth pooling layer of the fifth level.

[0107] For example, the output of S216 is used as the input of S217. S217 performs an upsampling operation, specifically by spatially replicating each element in the feature map output by the fifth pooling layer to increase the size of the feature map. The upsampling operation is performed so that the feature map is expanded from a smaller size (such as the size of S216 is 15x15) to a size of 30x30, while keeping the number of channels unchanged, that is, a feature map of size 30x30x256 is obtained.

[0108] S218, concatenating the upsampled feature map with the feature map extracted at the fourth level.

[0109] For example, the output of S217 is used as the input of S218. The feature map (30x30x256) output by S217 and the feature map (30x30x256) output by S213 are used as the input of S218 at the same time. In the S218 layer, these two feature maps with the same spatial size but containing different feature information are concatenated along the channel dimension through a concatenation operation. This concatenation operation increases the number of channels of the feature map, resulting in a feature map of size 30x30x512.

[0110] S219, performing an upsampling operation on the concatenated feature map.

[0111] For example, the output of S218 is used as the input of S219 layer. In S219, an upsampling operation is also performed to increase the spatial resolution of the feature map by copying each element in the feature map. This operation increases the size of the feature map from 30x30 to 60x60 while keeping the number of channels unchanged, resulting in a feature map of size 60x60x512.

[0112] S220, concatenating the feature map after the upsampling operation and the feature map extracted at the third level.

[0113] For example, the output feature map of S219 (60x60x512) and the output feature map of S210 (60x60x128) are used as the input of S220 layer. In S220, these two feature maps are concatenated by performing a concatenation operation to increase the number of channels of the feature map. The concatenation operation combines feature information from different levels to obtain a feature map of size 60x60x640.

[0114] S221, performing an upsampling operation on the concatenated feature map.

[0115] For example, the output of S220 is used as the input of S221 layer. In S221, an upsampling operation is performed to increase the size of the feature map by copying each element in the feature map. This operation increases the feature map from 60x60 to 120x120 while keeping the number of channels unchanged, resulting in a feature map of size 120x120x640.

[0116] S222, concatenating the feature map after the upsampling operation with the feature map extracted at the second level.

[0117] For example, the output feature map of S221 (120x120x640) and the output feature map of S207 layer (120x120x64) are used as the input of S222. In S222, the two feature maps are concatenated by performing a concatenation operation to increase the number of channels of the feature map. This concatenation operation combines feature information from different levels to obtain a feature map of size 120x120x704.

[0118] S223, performing an upsampling operation on the concatenated feature map.

[0119] For example, the output of S222 is used as the input of S223. In S223, an upsampling operation is performed to increase the size of the feature map by copying each element in the feature map. This operation increases the feature map from 120x120 to 240x240 while keeping the number of channels unchanged, resulting in a feature map of size 240x240x704.

[0120] S224, concatenating the feature map after the upsampling operation with the feature map extracted at the first level to obtain a fused feature map.

[0121] For example, the output feature map of S223 (240x240x704) and the output feature map of S204 (240x240x32) are used as the input of S224. In S224, the two feature maps are concatenated by performing a concatenation operation to increase the number of channels of the feature map. This concatenation operation combines feature information from different levels to obtain a feature map of size 240x240x736.

[0122] In the above S217 to S224, it is demonstrated how to increase the spatial resolution and number of channels of the feature map by performing upsampling and splicing operations in the feature fusion module of the neural network, so that the neural network can fuse multi-level feature information while maintaining spatial details.

[0123] S225, sending the fused feature map to the output module, performing an upsampling operation on the fused feature map, and generating a matching feature map.

[0124] For example, S225 performs an upsampling operation on the fused feature map output by S224, and uses the element replication method to double the size of the feature map from 240x240 to 480x480, while the number of channels remains unchanged, and finally obtains a feature map of 480x480x736, so that the spatial resolution of the matching feature map matches the spatial resolution of the input image in S202, that is, both are 480x480.

[0125] S226, performing a convolution operation on the matching feature map to generate a single-channel feature map.

[0126] For example, S226 uses a 1x1x736 convolution kernel to perform a convolution operation on the matching feature map output by S225. Since the spatial size of the convolution kernel is 1x1 and the stride is 1, the spatial resolution of the matching feature map remains unchanged, that is, 480x480. Since there is only one convolution kernel and its depth is the same as the number of channels of the input matching feature map, the convolution operation reduces the number of channels of the output feature map from 736 to 1, and finally obtains a single-channel feature map of 480x480x1.

[0127] S227, post-processing the single channel feature map.

[0128] Exemplarily, by analyzing the single-channel feature map output by S226, it is determined whether each pixel point belongs to the text area, and the determination results are summarized to determine the specific position of the text in the input image.

[0129] Specifically, each pixel value in the single-channel feature map can be analyzed and compared with a preset threshold to determine which pixels in the input image belong to the text area. If the pixel value is higher than the threshold, the pixel is marked as part of the text area. By summarizing all these marks, the specific location of the text in the input image can be located and identified.

[0130] For example, the size of the single-channel feature map output by S226 is 480x480, and each pixel value represents the feature intensity of the corresponding position in the input image. Assuming the preset threshold is 0.5, if the value of a pixel is 0.7, which is higher than the threshold 0.5, the position in the input image corresponding to this pixel is marked as part of the text area. If the value of another pixel is 0.3, which is lower than the threshold, it is not marked as a text area. By performing such an inspection on the entire single-channel feature map, a binary image can be obtained, in which white pixels represent text areas and black pixels represent non-text areas. This binary image shows the location information of the text area in the input image.

[0131] Since S226 outputs a single-channel privilege map, each pixel has a clear value to indicate whether it belongs to the text area, so this information can be fused through logical operations (such as threshold determination) to obtain the precise location of the text area. The output of the single-channel privilege map reduces the number of feature maps that need to be processed, thereby reducing the amount of calculation.

[0132] The image processing device provided by the present application is described below. The image processing device described below and the image processing method described above can be referenced to each other.

[0133] Please refer to Figure 5 , Figure 5 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. An image processing device 500 includes a receiving module 510 and a neural network 520.

[0134] Exemplarily, the receiving module 510 is used to obtain an input image to be processed.

[0135] Exemplarily, the neural network 520 includes a feature extraction module, a feature fusion module and an output module; the feature extraction module is responsible for extracting features from the input image layer by layer and generating a feature map; the feature fusion module obtains a fused feature map by performing upsampling and splicing operations on the feature map to fuse feature maps from different levels; the output module first upsamples the fused feature map to adjust its spatial resolution to match the input image, and then generates a matching feature map; finally, the output module performs a convolution operation on the matching feature map to generate a single-channel feature map, which is used to identify whether there is a text area in the input image.

[0136] Exemplarily, the receiving module 510 is further configured to:

[0137] The input image is preprocessed to adjust the input image to a preset size and a preset number of channels.

[0138] Exemplarily, the feature extraction module includes multiple convolutional layers and multiple pooling layers; wherein the multiple convolutional layers are used to gradually increase the number of channels of the feature map and keep the spatial resolution of the feature map constant, and the multiple pooling layers are used to reduce the spatial resolution of the feature map layer by layer.

[0139] Exemplarily, the neural network 520 is also used to:

[0140] Performing an upsampling operation on the feature map extracted by the final level of the feature extraction module;

[0141] Concatenate the upsampled feature map with the feature map extracted from the next previous level;

[0142] The concatenated feature map is then upsampled and concatenated with the feature map extracted from the previous level.

[0143] This process is repeated until the feature map extracted at the initial level is processed, and finally the fused feature map that fuses feature information of all levels is obtained.

[0144] Exemplarily, the neural network structure includes multiple layers, each layer consists of several convolutional layers and at least one pooling layer; the convolutional layers and pooling layers in each layer work together to extract features at different levels of the input image; the layers are arranged from front to back, the initial layer is responsible for extracting low-level features, and the subsequent layers are responsible for extracting higher-level features; the final layer is responsible for outputting the results after network processing, and the initial layer receives the input image.

[0145] Exemplarily, in the neural network, the input of each layer first passes through a convolutional layer, the output feature map of the convolutional layer serves as the input of the next convolutional layer, the output of the second convolutional layer serves as the input of the pooling layer, and the output of the pooling layer is passed to the first convolutional layer of the next layer for step-by-step feature extraction, and each layer further extracts features based on the previous layer.

[0146] Exemplarily, in the neural network, the processed output of each convolutional layer is connected to a preset activation function to perform nonlinear transformation on the feature map output by the convolutional layer.

[0147] Exemplarily, the image processing apparatus 500 is further configured to:

[0148] Analyzing each feature point in the single-channel feature map to determine whether they indicate that the corresponding pixel point in the input image belongs to a text area;

[0149] For each feature point, if its value is higher than a preset threshold, it is determined that the pixel point in the input image corresponding to the feature point belongs to the text area;

[0150] The determination results of all feature points are integrated, and the position information of the text area of ​​the input image is identified by counting the positions of the feature points above the threshold in the single-channel feature map.

[0151] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 730 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the speech recognition method.

[0152] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0153] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the speech recognition method provided by the above methods.

[0154] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the speech recognition methods provided above.

[0155] The embodiments of the present application provide an electronic device, a computer program product, and a processor-readable storage medium, on which the computer program stored enables the processor to implement all the method steps implemented in the above-mentioned method embodiments and to achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiments will not be described in detail herein.

[0156] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: Applied to an image processing device, the image processing device includes a neural network, the neural network includes a feature extraction module, a feature fusion module and an output module, and the image processing method includes: Get the input image to be processed; Sending the input image to the feature extraction module to extract features from the input image layer by layer to generate a feature map; The feature map is sent to the feature fusion module, and the feature maps from different levels are fused by performing upsampling and concatenation operations on the feature map to obtain a fused feature map; Sending the fused feature map to the output module, performing an upsampling operation on the fused feature map to adjust its spatial resolution so that it matches the spatial resolution of the input image, and generating a matching feature map; and A convolution operation is performed on the matching feature map to generate a single-channel feature map, where the single-channel feature map is used to identify whether a text area exists in the input image.

2. The image processing method according to claim 1, characterized in that: After obtaining the input image to be processed, the image processing method further includes: The input image is preprocessed to adjust the input image to a preset size and a preset number of channels.

3. The image processing method according to claim 1, characterized in that: The feature extraction module includes multiple convolutional layers and multiple pooling layers; wherein the multiple convolutional layers are used to gradually increase the number of channels of the feature map and keep the spatial resolution of the feature map constant, and the multiple pooling layers are used to reduce the spatial resolution of the feature map layer by layer.

4. The image processing method according to claim 3, characterized in that: The step of sending the feature map to the feature fusion module, and fusing feature maps from different levels by performing upsampling and concatenation operations on the feature map to obtain a fused feature map comprises: Performing an upsampling operation on the feature map extracted by the final level of the feature extraction module; Concatenate the upsampled feature map with the feature map extracted from the next previous level; The concatenated feature map is then upsampled and concatenated with the feature map extracted from the previous level. This process is repeated until the feature map extracted at the initial level is processed, and finally the fused feature map that fuses feature information of all levels is obtained.

5. The image processing method according to claim 4, characterized in that: The neural network structure includes multiple levels, each level is composed of several convolutional layers and at least one pooling layer; the convolutional layers and pooling layers in each level work together to extract features of different levels of the input image; the levels are arranged from front to back, the initial level is responsible for extracting low-level features, and the subsequent levels are responsible for extracting higher-level features; the final level is responsible for outputting the results after network processing, and the initial level receives the input image.

6. The image processing method according to claim 5, characterized in that: In the neural network, the input of each layer first passes through a convolutional layer, and the output feature map of the convolutional layer is used as the input of the next convolutional layer. The output of the second convolutional layer is then used as the input of the pooling layer, and the output of the pooling layer is passed to the first convolutional layer of the next layer for gradual feature extraction. Each layer further extracts features based on the previous layer.

7. The image processing method according to claim 3, characterized in that: In the neural network, the processed output of each convolutional layer is connected to a preset activation function to perform nonlinear transformation on the feature map output by the convolutional layer.

8. The image processing method according to claim 1, characterized in that: After performing the step of performing a convolution operation on the matching feature map to generate a single-channel feature map, the image processing method further includes: Analyzing each feature point in the single-channel feature map to determine whether they indicate that the corresponding pixel point in the input image belongs to a text area; For each feature point, if its value is higher than a preset threshold, it is determined that the pixel point in the input image corresponding to the feature point belongs to the text area; The determination results of all feature points are integrated, and the position information of the text area of ​​the input image is identified by counting the positions of the feature points above the threshold in the single-channel feature map.

9. An image processing device, characterized in that: The image processing device comprises: A receiving module, used for obtaining an input image to be processed; A neural network, comprising a feature extraction module, a feature fusion module and an output module; the feature extraction module is responsible for extracting features from the input image layer by layer and generating a feature map; the feature fusion module obtains a fused feature map by performing upsampling and splicing operations on the feature map to fuse feature maps from different layers; the output module first upsamples the fused feature map to adjust its spatial resolution to match the input image, thereby generating a matching feature map; finally, the output module performs a convolution operation on the matching feature map to generate a single-channel feature map, which is used to identify whether there is a text area in the input image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

Images