Image processing method and device, electronic equipment and readable storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and readable storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a target image and an image processing model; inputting the target image into the image processing model, performing down-sampling processing on the target image by the image processing model, and obtaining a first image feature of the target image; performing global feature extraction on the first image feature, and obtaining a global feature of the target image; performing local feature extraction on the first image feature, and obtaining a plurality of local features of the target image; and performing image processing on the target image based on the global feature and the plurality of local features, and obtaining an image processing result of the target image. Through the down-sampling processing on the target image, the calculation amount of subsequent processing is reduced. Through the global feature and the local feature of the first image feature, double-scale image features are extracted, so that the accuracy and efficiency of image processing are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to an image processing method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] In the field of computer technology, an image processing model is often used to process a target image. Generally, the target image is input into the image processing model, and the image processing model extracts image features of the target image and processes the target image based on the image features of the target image.

[0003] In related technologies, when extracting image features of a target image, an image processing model can extract global features of the target image based on an attention mechanism to obtain the image features of the target image. Since global features of the target image are extracted, the amount of calculation is large, and the image features of the target image obtained by this feature extraction method have poor representation ability, which leads to low accuracy of the image processing result obtained when the target image is processed based on the image features of the target image. SUMMARY

[0004] The present application provides an image processing method and device, electronic equipment and readable storage medium, which can be used to solve the problem of large amount of calculation and low accuracy of image processing result in related technologies. The technical solution includes the following contents.

[0005] In one aspect, an image processing method is provided, which includes:

[0006] obtaining a target image and an image processing model;

[0007] inputting the target image into the image processing model, performing down-sampling processing on the target image by the image processing model to obtain first image features of the target image;

[0008] performing global feature extraction on the first image features to obtain global features of the target image;

[0009] performing local feature extraction on the first image features to obtain a plurality of local features of the target image;

[0010] processing the target image based on the global features and the plurality of local features to obtain an image processing result of the target image.

[0011] In another aspect, an image processing device is provided, which includes:

[0012] an obtaining module configured to obtain a target image and an image processing model;

[0013] The downsampling processing module is configured to input the target image into the image processing model, perform downsampling processing on the target image by using the image processing model, and obtain a first image feature of the target image.

[0014] The global feature extraction module is configured to perform global feature extraction on the first image feature, and obtain a global feature of the target image.

[0015] The local feature extraction module is configured to perform local feature extraction on the first image feature, and obtain a plurality of local features of the target image.

[0016] The image processing module is configured to perform image processing on the target image based on the global feature and the plurality of local features, and obtain an image processing result of the target image.

[0017] In a possible implementation, the global feature extraction module is configured to perform downsampling processing on the first image feature, and obtain a second image feature of the target image; and perform global feature extraction on the second image feature, and obtain a global feature of the target image.

[0018] In a possible implementation, the local feature extraction module is configured to perform block processing on the first image feature, and obtain a plurality of feature blocks; and for any feature block, perform feature extraction on the any feature block, and obtain a local feature of the target image.

[0019] In a possible implementation, the image processing module is configured to determine a third image feature of the target image based on the global feature and the plurality of local features; determine a target image feature of the target image based on the third image feature; and perform image processing on the target image based on the target image feature, and obtain an image processing result of the target image.

[0020] In a possible implementation, the image processing module is configured to splice the global feature and the plurality of local features, and obtain a first spliced feature of the target image; perform convolution processing on the first spliced feature at at least two scales, and obtain features corresponding to respective scales; splice the features corresponding to the respective scales, and obtain a second spliced feature of the target image; and determine a third image feature of the target image based on the second spliced feature.

[0021] In a possible implementation, the image processing module is configured to: in response to the first loop end condition being met and the second loop end condition being met, take the third image feature as the target image feature; in response to the second loop end condition not being met, take the third image feature as a first image feature of a next loop, and perform a loop starting from the global feature extraction on the first image feature to obtain a global feature of the target image until the first loop end condition is met and the second loop end condition is met to obtain the target image feature; and in response to the first loop end condition not being met but the second loop end condition being met, take the third image feature as a target image of a next loop, and perform a loop starting from the downsampling processing on the target image to obtain a first image feature of the target image until the first loop end condition is met and the second loop end condition is met to obtain the target image feature.

[0022] In a possible implementation, the image processing model comprises at least one feature extraction network layer in series, and each feature extraction network layer comprises at least one feature extraction network block in series; the first loop end condition is met when the current loop is a loop in a last feature extraction network layer of the at least one feature extraction network layer in series; and the second loop end condition is met when the current loop is a loop in a last feature extraction network block of the at least one feature extraction network block in series comprised in the feature extraction network layer.

[0023] In another aspect, an electronic device is provided, which comprises a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement any of the above-described image processing methods.

[0024] In another aspect, a computer readable storage medium is also provided, which stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement any of the above-described image processing methods.

[0025] In another aspect, a computer program or computer program product is also provided, which stores at least one computer program, which is loaded and executed by a processor to enable an electronic device to implement any of the above-described image processing methods.

[0026] The technical solutions provided in the present application bring at least the following beneficial effects:

[0027] The technical scheme provided in the application first performs downsampling processing on a target image to obtain first image features, and the amount of calculation in subsequent processing is reduced through the downsampling processing. Then, global feature extraction is performed on the first image features to obtain global features, and local feature extraction is performed on the first image features to obtain a plurality of local features, so that double-scale image features are extracted, and the accuracy of image processing on the target image based on the global features and the plurality of local features can be improved. Since the amount of calculation is low, the efficiency of image processing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 is a schematic diagram of an implementation environment of an image processing method provided by an embodiment of the present application;

[0030] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0031] Figure 3 is a schematic diagram of a DSSA block provided by an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of a DST block provided by an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of a DSFFN block provided by an embodiment of the present application;

[0034] Figure 6 is a schematic diagram of an image processing model provided by an embodiment of the present application;

[0035] Figure 7 is a flowchart of an image processing method provided by an embodiment of the present application;

[0036] Figure 8 is a schematic diagram of a receptive field provided by an embodiment of the present application;

[0037] Figure 9 is a schematic diagram of the relationship between predicted performance and amount of calculation provided by an embodiment of the present application;

[0038] Figure 10 is a structural schematic diagram of an image processing device provided by an embodiment of the present application;

[0039] Figure 11 is a structural schematic diagram of a terminal device provided by an embodiment of the present application.

[0040] Figure 12 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0042] Figure 1 is an implementation environment schematic diagram of an image processing method provided by an embodiment of the present application, as shown in the figure, the implementation environment includes a terminal device 101 and a server 102. Wherein, the image processing method in the embodiments of the present application can be executed by the terminal device 101, or can be executed by the server 102, or can be executed by the terminal device 101 and the server 102 together. Figure 1

[0043] The terminal device 101 can be a smart phone, a game console, a desktop computer, a tablet computer, a laptop computer, a smart television, a smart vehicle device, a smart voice interaction device, a smart home appliance, etc. The server 102 can be a server, or a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 can be connected with the terminal device 101 through a wired network or a wireless network. The server 102 can have functions of data processing, data storage and data transceiving, which are not limited in the embodiments of the present application. The number of the terminal device 101 and the server 102 is not limited, and can be one or more.

[0044] The image processing method provided by the embodiments of the present application can be realized based on artificial intelligence technology. Artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0045] ​Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0046] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, and intelligent transportation technologies.

[0047] Based on the above implementation environment, this application provides an image processing method to... Figure 2 The flowchart shown in this embodiment of the application illustrates an image processing method. This method can be implemented by... Figure 1 The method can be executed by either terminal device 101 or server 102, or jointly by both. For ease of description, the terminal device 101 or server 102 executing the image processing method in this embodiment is referred to as an electronic device, and the method can be executed by an electronic device. Figure 2 As shown, the method includes steps 201 to 205.

[0048] Step 201: Obtain the target image and the image processing model.

[0049] The target image in this application embodiment can be any image, such as a cell image, a remote sensing image, a photographic image, etc. The number of target images is at least one.

[0050] The embodiments of the present application do not limit the acquisition manner of the target image. Illustratively, the target image can be an image imported by the user from other devices, for example, the user imports a certain image in a universal serial bus (USB) flash disk into the electronic device, so that the electronic device acquires the image, that is, the electronic device acquires the target image. The target image can also be an image photographed by the user using the electronic device.

[0051] The embodiments of the present application can also acquire an image processing model. The embodiments of the present application do not limit the model structure and model size of the image processing model.

[0052] In step 202, the target image is input into the image processing model, and the target image is down-sampled by the image processing model to obtain a first image feature of the target image.

[0053] In the embodiments of the present application, the image processing model includes a down-sampling network layer. After the target image is input into the image processing model, the target image is down-sampled by the down-sampling network layer to obtain the first image feature of the target image.

[0054] The embodiments of the present application do not limit the structure and size of the down-sampling network layer. Illustratively, the down-sampling network layer can be a convolution layer with a stride s, where s is a positive integer greater than 1, such as s=2. Alternatively, the down-sampling network layer includes at least two convolution layers connected in series, the stride of any one of the convolution layers is a positive integer greater than 1, and the strides of any two of the convolution layers can be the same or different. Alternatively, the down-sampling network layer down-samples the target image by a pooling manner, in which case the down-sampling network layer can be a pooling layer. Alternatively, the down-sampling network layer can down-sample the target image by a bilinear interpolation or the like.

[0055] It should be noted that down-sampling the target image by the down-sampling network layer can reduce the scale of the feature, for example, when the down-sampling network layer is a convolution layer with a stride of 2, the scale of the feature can be reduced to one half of the original, thereby reducing the computational complexity.

[0056] In step 203, global feature extraction is performed on the first image feature to obtain a global feature of the target image.

[0057] In the embodiments of the present application, the down-sampling network layer of the image processing model can be spliced with a global feature extraction network layer. The global feature extraction network layer is used to perform global feature extraction on the first image feature to obtain the global feature of the target image.

[0058] Optionally, there is a layer normalization (LN) network layer between the down-sampling network layer and the global feature extraction network layer, and the LN network layer is configured to perform layer normalization on the first image feature to prevent gradient explosion and gradient disappearance. During back propagation, the gradient of each layer is calculated by multiplying the data (i.e., the feature) of the layer with the gradient passed from the previous layer to obtain the gradient of the layer. Through layer-by-layer accumulation, gradient explosion or disappearance may occur. By performing layer normalization on the first image feature through the LN network layer, the data of the layer is normalized to a range with a mean of zero and a variance of 1, so that the data distribution is consistent, and gradient explosion or disappearance is avoided.

[0059] In the embodiments of the present application, the LN network layer can be used to perform layer normalization on the first image feature to obtain the first image feature after layer normalization, and then the global feature extraction network layer can be used to perform global feature extraction on the first image feature after layer normalization to obtain the global feature of the target image.

[0060] In one possible implementation, the global feature extraction on the first image feature to obtain the global feature of the target image includes: performing down-sampling processing on the first image feature to obtain a second image feature of the target image; and performing global feature extraction on the second image feature to obtain the global feature of the target image.

[0061] In the embodiments of the present application, the global feature extraction network layer includes a down-sampling network layer and a global attention network layer connected in series. The down-sampling network layer is configured to perform down-sampling processing on the first image feature (or the first image feature after layer normalization) again to obtain the second image feature, so as to further reduce the scale of the feature and reduce the computational complexity. The global attention network layer is configured to perform global feature extraction on the second image feature to obtain the global feature.

[0062] Optionally, the down-sampling network layer in the global feature extraction network layer is a convolution layer with a step size of s, and the down-sampling processing on the first image feature can be implemented by performing convolution operation on the first image feature with a step size of s to obtain the second image feature. The first image feature can be denoted as X∈R H×W×C , and the second image feature can be denoted as wherein X is the first image feature, R is a real number, H is the height of the first image feature, W is the width of the first image feature, C is the number of channels of the first image feature, X G is the second image feature, and s is the step size of the convolution operation.

[0063] Optionally, the global attention network layer is a network layer based on a self-attention (SA) mechanism. When performing global feature extraction on the second image feature based on the network layer based on the SA mechanism, on one hand, based on the weight of the key vector, the second image feature is subjected to a convolution operation according to K G =conv(w K X G ), to obtain a key vector corresponding to the second image feature. On the other hand, based on the weight of the value vector, the second image feature is subjected to a convolution operation according to V G =conv(w v X G ), to obtain a value vector corresponding to the second image feature. On the other hand, based on the weight of the query vector, the first image feature (or the first image feature after layer normalization processing) is subjected to a linear transformation according to Q G =w Q X, to obtain a query vector corresponding to the second image feature.

[0064] wherein K G is the key vector corresponding to the second image feature, conv represents a convolution operation, w K is the weight of the key vector, X G is the second image feature, V G is the value vector corresponding to the second image feature, w v is the weight of the value vector, Q G is the query vector corresponding to the second image feature, w Q is the weight of the query vector, and X is the first image feature. Wherein Q G has the same dimension as X, therefore, Q G ∈R H×W×C , and K G , V G have the same dimension as X G , therefore, K G , R is a real number, H is the height of the first image feature, W is the width of the first image feature, C is the channel number of the first image feature, and s is the step length of the convolution operation.

[0065] Next, based on the key vector corresponding to the second image feature, the value vector corresponding to the second image feature, and the query vector corresponding to the second image feature, attention calculation is performed according to the formula (1) as shown below, to determine the global feature.

[0066]

[0067] wherein A G is the global feature, softmax is a regression function, Q G is the query vector corresponding to the second image feature, KG K is a key vector corresponding to the second image feature, T is a transpose matrix symbol, C is a channel number of the first image feature, is a root symbol, V G is a value vector corresponding to the second image feature. Wherein, A G is the same as the dimension of the first image feature X, therefore, A G ∈R H×W×C , R is a real number, H is the height of the first image feature, W is the width of the first image feature, and C is the channel number of the first image feature.

[0068] Since the key vector K G corresponding to the second image feature and the value vector K G corresponding to the second image feature are determined based on the second image feature obtained by performing down-sampling processing on the first image feature. Therefore, based on K G , V G , the coarse-grained global feature can be determined according to formula (1), and the computational complexity is O(2C(HW) 2 / s 2 ), if K G , V G are determined based on the first image feature, that is, the global feature extraction is directly performed on the first image feature, then the computational complexity is O(2C(HW) 2 ). Wherein, O is a computational complexity symbol, H is the height of the first image feature, W is the width of the first image feature, C is the channel number of the first image feature, and s is the step length of the convolution operation. By first performing down-sampling processing on the first image feature, and then performing global feature extraction on the second image feature obtained by the down-sampling processing, the computational complexity can be reduced by s 2 times, thereby improving the operation speed.

[0069] In step 204, local feature extraction is performed on the first image feature to obtain a plurality of local features of the target image.

[0070] In the embodiment of the application, the down-sampling network layer of the image processing model can be spliced with a local feature extraction network layer. The local feature extraction network layer is used to perform local feature extraction on the first image feature to obtain a plurality of local features of the target image. Wherein, the local feature extraction network layer and the global feature extraction network layer are connected in parallel after the down-sampling network layer of the image processing model.

[0071] Optionally, there is an LN network layer between the down-sampling network layer and the local feature extraction network layer, and the LN network layer is used to perform layer normalization processing on the first image feature to prevent gradient explosion and gradient disappearance.

[0072] The LN network layer is between the downsampling network layer and the local feature extraction network layer, and the LN network layer is between the downsampling network layer and the global feature extraction network layer. The two LN network layers can be the same, that is, the LN network layer is connected in series after the downsampling network layer, and the local feature extraction network layer and the global feature extraction network layer are connected in parallel after the LN network layer. The two LN network layers can also be two different LN network layers, that is, two branches are connected in parallel after the downsampling network layer, one branch includes the LN network layer and the global feature extraction network layer connected in series, and the other branch includes the LN network layer and the local feature extraction network layer connected in series.

[0073] It should be noted that when the local feature extraction network layer and the global feature extraction network layer are connected in series after the downsampling network layer of the image processing model, the local feature of the image is first focused and then the global feature of the image is focused, and the local feature of the image and the global feature of the image cannot be effectively integrated. Similarly, when the global feature extraction network layer and the local feature extraction network layer are connected in series after the downsampling network layer of the image processing model, the global feature of the image is first focused and then the local feature of the image is focused, and the local feature of the image and the global feature of the image cannot be effectively integrated. In addition, under normal circumstances, some network layers such as multi-layer perceptron (MLP) layers are spliced after the local feature extraction network layer and the global feature extraction network layer, and this series connection mode will cause the calculation amount of the model to be redundant and the calculation efficiency to be low.

[0074] In the embodiment of the present application, the local feature extraction network layer and the global feature extraction network layer are connected in parallel after the downsampling network layer of the image processing model, which can effectively integrate the local feature of the image and the global feature of the image. In addition, if network layers are spliced after the local feature extraction network layer and the global feature extraction network layer, the number of network layers can be reduced, the calculation amount can be reduced, and the calculation efficiency can be improved.

[0075] In the embodiment of the present application, the LN network layer can be used to perform layer normalization processing on the first image feature to obtain a layer-normalized first image feature, and then the local feature extraction network layer can be used to perform local feature extraction on the layer-normalized first image feature to obtain a plurality of local features of the target image.

[0076] In a possible implementation, the local feature extraction is performed on the first image feature to obtain a plurality of local features of the target image, including: performing block processing on the first image feature to obtain a plurality of feature blocks; and for any feature block, performing feature extraction on the feature block to obtain a local feature of the target image.

[0077] In the embodiments of the present application, the first image feature (or the first image feature after layer normalization processing) can be subjected to block processing to obtain a plurality of feature blocks of the same size. The size of the feature block is denoted as w x w, where w is a positive integer. In the foregoing, the first image feature is denoted as X e R H×W×C The first image feature after reshaping is denoted as where X L is the first image feature after reshaping. It can be seen from the first image feature after reshaping that the first image feature is reshaped, that is, the first image feature is divided into a plurality of w x w feature blocks.

[0078] Optionally, the local feature extraction network layer is a network layer based on the SA mechanism. The network layer based on the SA mechanism can perform feature extraction on any feature block to obtain a local feature. That is, for the first image feature after reshaping, the network layer based on the SA mechanism can perform attention calculation inside each w x w feature block to obtain a local feature corresponding to each feature block.

[0079] First, based on the weight of the query vector, the Q L = w Q X L is linearly transformed on the first image feature after reshaping to obtain the query vector corresponding to the first image feature. Based on the weight of the key vector, the K L = w K X L is linearly transformed on the first image feature after reshaping to obtain the key vector corresponding to the first image feature. Based on the weight of the value vector, the V L = w v X L is linearly transformed on the first image feature after reshaping to obtain the value vector corresponding to the first image feature.

[0080] where Q L is the query vector corresponding to the first image feature, w Q is the weight of the query vector, X L is the first image feature after reshaping, K L is the key vector corresponding to the first image feature, w K is the weight of the key vector, V L is the value vector corresponding to the first image feature, and w v is the weight of the value vector. Q L , K L , V L and X L have the same dimension, so Q L , K L ,

[0081] Next, based on the key vector corresponding to the first image feature, the value vector corresponding to the first image feature and the query vector corresponding to the first image feature, attention calculation is performed inside each feature block according to formula (2) as shown below to obtain the local feature corresponding to each feature block.

[0082]

[0083] wherein, A L includes the local feature corresponding to each feature block, reshape represents a reshaping operation, softmax is a regression function, Q L is the query vector corresponding to the first image feature, K L is the key vector corresponding to the first image feature, T is the symbol of a transposed matrix, C is the channel number of the first image feature, is the root symbol, V L is the value vector corresponding to the first image feature. Wherein, A L is the same as the dimension of the first image feature X, therefore, A L ∈R H×W×C , R is a real number, H is the height of the first image feature, W is the width of the first image feature, and C is the channel number of the first image feature.

[0084] Since the embodiment of the present application performs attention calculation inside each feature block, the embodiment of the present application extracts the local feature corresponding to each feature block. The calculation complexity is O(2CHWw 2 ), compared with the calculation complexity O(2C(HW) 2 ) corresponding to the direct global feature extraction of the first image feature, the calculation complexity is reduced from quadratic to linear, and the operation speed is improved.

[0085] In step 205, based on the global feature and the plurality of local features, the target image is processed to obtain an image processing result of the target image.

[0086] In the embodiment of the present application, the image processing model can determine a target image feature of the target image based on the global feature and the plurality of local features, wherein the target image feature is used to represent the texture information of the target image. The image processing model can process the target image based on the target image feature to obtain an image processing result of the target image. The image processing includes but is not limited to image classification processing, image segmentation processing, image detection processing, etc.

[0087] For example, when the target image is an image related to an industrial product (such as a glass product, a circuit board, a dashboard, etc.), the image processing model can determine a target image feature of the target image, and perform image detection processing on the target image based on the target image feature, to detect defects in the industrial product.

[0088] When the target image is a road image, the image processing model can determine a road image feature of the road image, and perform image segmentation processing on the road image based on the road image feature, to segment objects such as roads, buildings, vehicles, and trees in the road image, thereby realizing automatic driving, etc.

[0089] When the target image is a medical image, the image processing model can determine a medical image feature of the medical image, and perform image classification processing on the medical image based on the medical image feature, to analyze lesion types in the medical image, etc.

[0090] In a possible implementation, the image processing model performs image processing on the target image based on the global feature and the plurality of local features, to obtain an image processing result of the target image, including: determining a third image feature of the target image based on the global feature and the plurality of local features; determining a target image feature of the target image based on the third image feature; and performing image processing on the target image based on the target image feature, to obtain the image processing result of the target image.

[0091] In the embodiments of the present application, since the local feature extraction network layer and the global feature extraction network layer are connected in parallel, the global feature and the plurality of local features need to be fused to obtain a third image feature of the target image.

[0092] Optionally, the determining of the third image feature of the target image based on the global feature and the plurality of local features includes: splicing the global feature and the plurality of local features to obtain a first spliced feature of the target image; performing convolution processing on the first spliced feature at least two scales to obtain features corresponding to the scales; splicing the features corresponding to the scales to obtain a second spliced feature of the target image; and determining the third image feature of the target image based on the second spliced feature.

[0093] The image processing model includes a dual-scale self-attention mechanism (DSSA) block, which includes the local feature extraction network layer and the global feature extraction network layer mentioned above. The DSSA block is connected in series after the downsampling network layer mentioned above, or the DSSA block is connected in series after the downsampling network layer and the LN network layer mentioned above. The DSSA block can splice the global feature and the plurality of local features to obtain a first spliced feature of the target image.

[0094] The embodiment of the application splices the global feature and the plurality of local features according to formula (3) shown as follows, to obtain the first spliced feature of the target image.

[0095] A = w Q × Concat(A G ,A L ) formula (3)

[0096] Wherein, A represents the first spliced feature of the target image. w Q is the weight corresponding to the splicing operation. Concat represents the splicing operation. A L includes the local feature corresponding to each feature block, that is, A L includes the plurality of local features. A G is the global feature.

[0097] According to formula (3), A G and A L are spliced in the channel dimension to obtain A. Wherein, A ∈ R H×W×2C , R is a real number, H is the height of the first image feature, W is the width of the first image feature, and C is the channel number of the first image feature. Optionally, the channel compression is performed on A by linear operation, so as to compress the channel dimension of A from 2C to C, that is, the channel dimension of the compressed A is C.

[0098] Please refer to Figure 3 , Figure 3 is a schematic diagram of a DSSA block provided by the embodiment of the application, which includes a local feature extraction network layer and a global feature extraction network layer in parallel. After the first image feature is input into the DSSA block, the local feature extraction network layer and the global feature extraction network layer respectively perform feature extraction on the first image feature.

[0099] Wherein, the local feature extraction network layer first blocks the first image feature to obtain a plurality of feature blocks. For each feature block (such as Figure 3 one feature block marked in the figure), feature extraction is performed on the one feature block to obtain the local feature corresponding to the one feature block. In this way, the local feature corresponding to each feature block can be obtained. The global feature extraction network layer first performs down-sampling processing on the first image feature to obtain a second image feature, and then performs global feature extraction on the second image feature to obtain a global feature. Then, the DSSA block splices the global feature and the local feature corresponding to each feature block to obtain the first spliced feature.

[0100] Optionally, the DSSA block can adopt a residual structure to facilitate better gradient propagation. At this time, the DSSA block can adopt a residual structure, update the first stitching feature based on the first image feature to obtain an updated first stitching feature, which can be simply referred to as the first stitching feature. An LN network layer can be connected in series after the DSSA block to perform layer normalization processing on the (updated or unupdated) first stitching feature to prevent gradient explosion and gradient disappearance, and obtain a layer-normalized first stitching feature.

[0101] In the embodiments of the present application, a Dual-Scaled Feedforward Neural Network (DSFFN) block can be connected in series after the DSSA block, and the DSFFN block determines the third image feature based on the first stitching feature. Alternatively, the DSFFN block can be connected in series after the connected DSSA block and LN network layer, and the DSFFN block determines the third image feature based on the layer-normalized first stitching feature.

[0102] Optionally, the LN network layer, the DSSA block, the LN network layer and the DSFFN block connected in series are taken as a Dual-Scaled Transformer (DST) block. Please refer to Figure 4 , Figure 4 is a schematic diagram of a DST block provided by the embodiments of the present application.

[0103] In the embodiments of the present application, after the first image feature is input into the DST block, the first image feature is subjected to layer normalization processing by the first LN network layer to obtain a layer-normalized first image feature, and the layer-normalized first image feature is subjected to feature extraction by the DSSA block to obtain the first stitching feature. Since the DSSA block adopts a residual structure, the first stitching feature needs to be updated based on the first image feature to obtain an updated first stitching feature.

[0104] Then, the updated first stitching feature is subjected to layer normalization processing by the second LN network layer to obtain a layer-normalized first stitching feature, and the DSFFN block determines the third image feature based on the layer-normalized first stitching feature. Since the DSFFN block adopts a residual structure, the third image feature needs to be updated based on the updated first stitching feature to obtain an updated third image feature. At this time, the DST block outputs the updated third image feature.

[0105] It should be noted that the DST block can not contain any LN network layer, or can not contain two LN network layers. The DSSA block in the DST block can not adopt a residual structure, and similarly, the DSFFN block in the DST block can not adopt a residual structure, which is not limited herein.

[0106] In the embodiments of the present application, the DSFFN block can determine the third image feature based on the first stitching feature (or the first stitching feature after layer normalization processing). The DSFFN block includes at least two scales of convolution layers, and each scale of convolution layer is used to perform convolution processing of the first stitching feature at the scale to obtain a feature corresponding to the scale. In the embodiments of the present application, the scale of the convolution layer is not limited. For example, the scale of the convolution layer is 1x1, that is, the convolution layer is a 1x1 convolution layer, and at this time, the 1x1 convolution layer performs 1x1 convolution processing on the first stitching feature to obtain a 1x1 corresponding feature. Alternatively, the scale of the convolution layer is 3x3, that is, the convolution layer is a 3x3 convolution layer, and at this time, the 3x3 convolution layer performs 3x3 convolution processing on the first stitching feature to obtain a 3x3 corresponding feature.

[0107] It should be noted that when the DSFFN block performs convolution processing of the first stitching feature at at least two scales, the DSFFN block can first perform fusion processing on the first stitching feature to better fuse the global feature and each local feature.

[0108] Optionally, the DSFFN block includes a fusion layer, and the fusion layer includes at least one of a convolution layer, a nonlinear mapping layer, and a batch normalization (BN) layer, and the connection mode of the convolution layer, the nonlinear mapping layer, and the batch normalization layer is not limited in the embodiments of the present application. The convolution layer can be a 1x1 convolution layer, which is used to perform convolution processing on the first stitching feature to fuse the global feature and each local feature. The nonlinear mapping layer can be a network layer based on Gaussian error linear unit (GELU), which is used to perform nonlinear mapping processing on the first stitching feature. The batch normalization layer is used to perform batch normalization processing on the first stitching feature.

[0109] After the fusion processing (that is, at least one of the convolution processing, the nonlinear mapping processing, and the batch normalization processing) on the first stitching feature is performed, the first stitching feature after the fusion processing is obtained, which can be referred to as the first stitching feature. Then, the first stitching feature is subjected to convolution processing at at least two scales to obtain features corresponding to each scale. The features corresponding to each scale are stitched to obtain the second stitching feature of the target image. Next, the second stitching feature is taken as the third image feature of the target image, or at least one of convolution processing, nonlinear mapping processing, and batch normalization processing is performed on the second stitching feature by using the fusion layer to realize fusion processing on the second stitching feature, and the second stitching feature after the fusion processing is obtained, which can be taken as the third image feature.

[0110] Optionally, the DSFFN block adopts a residual structure, and the first splicing feature is used to update the second splicing feature (or the second splicing feature after fusion processing), to obtain updated second splicing feature. At this time, the updated second splicing feature can be used as the third image feature.

[0111] See Figure 5 , Figure 5 is a schematic diagram of a DSFFN block provided by an embodiment of the present application. In the embodiment of the present application, the DSFFN block includes a fusion layer, a 1x1 convolution layer and a 3x3 convolution layer connected in parallel, a splicing layer, and a fusion layer.

[0112] After the first splicing feature is input into the DSFFN block, the first splicing feature is fused by the first fusion layer to obtain the first splicing feature after fusion processing. The first splicing feature after fusion processing is subjected to 1x1 convolution processing by the 1x1 convolution layer to obtain the 1x1 corresponding feature, and is subjected to 3x3 convolution processing by the 3x3 convolution layer to obtain the 3x3 corresponding feature. Then, the 1x1 corresponding feature and the 3x3 corresponding feature are spliced to obtain the second splicing feature. After that, the second splicing feature is fused by the second fusion layer to obtain the second splicing feature after fusion processing.

[0113] Figure 5 The DSFFN block in adopts a residual structure, and therefore, the first splicing feature is used to update the second splicing feature after fusion processing, to obtain the updated second splicing feature. At this time, the updated second splicing feature is used as the third image feature.

[0114] Next, the target image feature of the target image is determined based on the third image feature. Optionally, based on the third image feature, the target image feature of the target image is determined, including: in response to satisfying the first loop end condition and satisfying the second loop end condition, the third image feature is taken as the target image feature; in response to not satisfying the second loop end condition, the third image feature is taken as the first image feature of the next loop, and the loop is started from the global feature extraction of the first image feature to obtain the global feature of the target image, until the first loop end condition is satisfied and the second loop end condition is satisfied, to obtain the target image feature; in response to not satisfying the first loop end condition but satisfying the second loop end condition, the third image feature is taken as the target image of the next loop, and the loop is started from the down-sampling processing of the target image to obtain the first image feature of the target image, until the first loop end condition is satisfied and the second loop end condition is satisfied, to obtain the target image feature.

[0115] In the embodiments of the present application, after obtaining the third image feature, it can be determined whether the first loop end condition is met and whether the second loop end condition is met. Based on the determination result, different loop operations are performed until the target image feature is obtained. Optionally, the image processing model comprises at least one feature extraction network layer in series, and each feature extraction network layer comprises at least one feature extraction network block in series; the first loop end condition is met when the current loop is a loop in the last feature extraction network layer in the at least one feature extraction network layer in series; and the second loop end condition is met when the current loop is a loop in the last feature extraction network block in the at least one feature extraction network block in series comprised by any feature extraction network layer.

[0116] In the embodiments of the present application, the image processing model comprises at least one feature extraction network layer in series. After the target image is input into the image processing model, the target image is subjected to feature extraction by the first feature extraction network layer. Since each feature extraction network layer comprises a down-sampling network layer and at least one feature extraction network block in series, and each feature extraction network block is the DST block mentioned above, the down-sampling network layer first performs down-sampling processing on the target image to obtain a first image feature. Then, the third image feature is determined by the first feature extraction network block based on the first image feature.

[0117] When the first loop end condition is met and the second loop end condition is met, that is, the current loop is a loop in the last feature extraction network layer in the at least one feature extraction network layer in series, and the current loop is a loop in the last feature extraction network block in the at least one feature extraction network block in series comprised by the last feature extraction network layer, that is, the third image feature is determined by the last feature extraction network block in the last feature extraction network layer in the image processing model, the third image feature is taken as the target image feature.

[0118] When the second loop end condition is not met, the third image feature is taken as the first image feature of the next loop regardless of whether the first loop end condition is met. That is, as long as the current loop is a loop in a non-last feature extraction network block in the at least one feature extraction network block in series comprised by any feature extraction network layer, regardless of whether the any feature extraction network layer is the last feature extraction network layer, the third image feature is taken as the first image feature of the next loop. That is, the third image feature is determined by a non-last feature extraction network block in any feature extraction network layer in the image processing model, and the third image feature is taken as the first image feature of the next loop.

[0119] After the third image feature is taken as the first image feature of the next loop, the loop starting from step 203 is executed until the first loop end condition is met and the second loop end condition is met. Wherein, the loop starting from step 203 is executed, that is, the processing of the DST block is executed. For example, the processing of the DST block includes: on the one hand, performing global feature extraction on the first image feature to obtain a global feature, and on the other hand, performing local feature extraction on the first image feature to obtain a plurality of local features of the target image. Then, the global feature and the plurality of local features are spliced to obtain a first spliced feature, and the first spliced feature is subjected to convolution processing of at least two scales to obtain features corresponding to each scale. Next, the features corresponding to each scale are spliced to obtain a second spliced feature of the target image. After that, based on the second spliced feature, the third image feature of the target image is determined.

[0120] When the first loop end condition is not met but the second loop end condition is met, that is, the current loop is not the loop in the last feature extraction network layer in the image processing model, but the current loop is the loop in the last feature extraction network block in any feature extraction network layer including at least one series of feature extraction network blocks, that is, the third image feature is determined by the last feature extraction network block in the non-last feature extraction network layer in the image processing model, the third image feature is taken as the target image of the next loop.

[0121] After the third image feature is taken as the target image of the next loop, the loop starting from step 202 is executed until the first loop end condition is met and the second loop end condition is met. Wherein, the loop starting from step 202 is executed, that is, the processing of the feature extraction network layer is executed. For example, the processing of the feature extraction network layer includes: performing down-sampling processing on the target image to obtain a first image feature and the processing of the DST block.

[0122] When the first loop end condition is met and the second loop end condition is met, the third image feature can be taken as the target image feature. After that, the target image is subjected to image processing based on the target image feature to obtain an image processing result of the target image.

[0123] Optionally, the image processing model includes an image processing network layer, and the image processing network layer is configured to perform image processing on the target image based on the target image feature. Wherein, when the image processing network layer is a classification layer, the classification layer is configured to perform image classification processing on the target image based on the target image feature. When the image processing network layer is an image segmentation network, the image segmentation network is configured to perform image segmentation processing on the target image based on the target image feature. When the image processing network layer is an image detection network, the image detection network is configured to perform image detection processing on the target image based on the target image feature.

[0124] Please refer toFigure 6 , Figure 6 is a schematic diagram of an image processing model provided by an embodiment of the present application. In the embodiment of the present application, the image processing model comprises at least two serially connected feature extraction network layers (for example, 4 serially connected feature extraction network layers) and an image processing network layer, and each feature extraction network layer comprises a down-sampling network layer and at least two serially connected DST blocks (denoted as Ni serially connected DST blocks, where Ni is a positive integer).

[0125] In the embodiment of the present application, after the target image is input into the image processing model, the down-sampling network layer in the first feature extraction network layer performs down-sampling processing on the target image to obtain a first image feature. Then, the first DST block in the first feature extraction network layer determines a third image feature based on the first image feature. Since the third image feature is determined by the non-last DST block in the first feature extraction network layer, the third image feature is taken as the first image feature in the next loop, and the second DST block in the first feature extraction network layer determines a third image feature based on the first image feature. In this way, the third image feature is determined by the last DST block in the first feature extraction network layer.

[0126] Since the third image feature is determined by the last feature extraction network block in the non-last feature extraction network layer in the image processing model, the third image feature is taken as the target image in the next loop, and the down-sampling network layer in the second feature extraction network layer performs down-sampling processing on the target image to obtain a first image feature. Then, the first DST block in the second feature extraction network layer determines a third image feature based on the first image feature. Since the third image feature is determined by the non-last DST block in the second feature extraction network layer, the third image feature is taken as the first image feature in the next loop, and the second DST block in the second feature extraction network layer determines a third image feature based on the first image feature. In this way, the third image feature is determined by the last DST block in the second feature extraction network layer.

[0127] In this way, the third image feature is determined by the last feature extraction network block in the last feature extraction network layer in the image processing model, and the third image feature is taken as the target image feature. The image processing network layer performs image processing on the target image based on the target image feature to obtain an image processing result of the target image.

[0128] For example, the image processing model includes 4 feature extraction network layers, the size of the target image is HxWxC, where H is the height of the target image, W is the width of the target image, and C is the channel number of the target image. After the target image is input into the first feature extraction network layer, the down-sampling network layer in the first feature extraction network layer can perform down-sampling processing on the target image to obtain a first image feature, and the size of the first image feature is and determine a third image feature based on the first image feature, where H is the height of the target image, W is the width of the target image, and C1 is the channel number of the first image feature. The third image feature can be used as the target image for the next cycle, and the down-sampling network layer in the second feature extraction network layer can perform down-sampling processing on the target image to obtain a first image feature, and the size of the first image feature is and determine a third image feature based on the first image feature, where H is the height of the target image, W is the width of the target image, and C2 is the channel number of the first image feature. The third image feature can be used as the target image for the next cycle, and the down-sampling network layer in the third feature extraction network layer can perform down-sampling processing on the target image to obtain a first image feature, and the size of the first image feature is and determine a third image feature based on the first image feature, where H is the height of the target image, W is the width of the target image, and C3 is the channel number of the first image feature. The third image feature can be used as the target image for the next cycle, and the down-sampling network layer in the fourth feature extraction network layer can perform down-sampling processing on the target image to obtain a first image feature, and the size of the first image feature is and determine a third image feature based on the first image feature, where H is the height of the target image, W is the width of the target image, and C4 is the channel number of the first image feature.

[0129] It should be noted that any two down-sampling network layers in the image processing model can be the same or different. Similarly, any two DST blocks in the image processing model can be the same or different. The feature extraction network layer in the image processing model can be used as a skeleton network in different models (such as ResNet model).

[0130] The above method first performs down-sampling processing on the target image to obtain a first image feature, and through the down-sampling processing, the computational load of subsequent processing is reduced. Then, the global feature extraction is performed on the first image feature to obtain a global feature, and the local feature extraction is performed on the first image feature to obtain a plurality of local features, which realizes the extraction of double-scale image features, so that the accuracy of image processing based on the global feature and the plurality of local features can be improved. Since the computational load is low, the efficiency of image processing is improved.

[0131] The image processing method is described from the perspective of method steps above, and the image processing method of the embodiments of the present application is further described below in combination with Figure 7 Figure 7 is a flowchart of an image processing method provided by the embodiments of the present application.

[0132] In the embodiments of the present application, a target image and an image processing model can be acquired, where the model structure of the image processing model is as shown in Figure 6 and will not be described herein again. The image processing model processes the target image as follows:

[0133] The target image is first down-sampled to obtain a first image feature. Global feature extraction is performed on the first image feature to obtain a global feature, and local feature extraction is performed on the first image feature to obtain a plurality of local features. The global feature and the plurality of local features are spliced to obtain a first spliced feature. Convolution processing of at least two scales is performed on the first spliced feature to obtain features corresponding to each scale. The features corresponding to each scale are spliced to obtain a second spliced feature. A third image feature is determined based on the second spliced feature.

[0134] When the first loop end condition is met and the second loop end condition is met, the third image feature is taken as a target image feature. Image processing is performed on the target image based on the target image feature to obtain an image processing result.

[0135] When the second loop end condition is not met, the third image feature is taken as a first image feature of the next loop, and a loop is executed starting from "global feature extraction is performed on the first image feature to obtain a global feature, and local feature extraction is performed on the first image feature to obtain a plurality of local features", until the first loop end condition is met and the second loop end condition is met. The third image feature is taken as a target image feature. Image processing is performed on the target image based on the target image feature to obtain an image processing result.

[0136] When the first loop end condition is not met but the second loop end condition is met, the third image feature is taken as a target image of the next loop, and a loop is executed starting from "down-sampling processing is performed on the target image to obtain a first image feature", until the first loop end condition is met and the second loop end condition is met. The third image feature is taken as a target image feature. Image processing is performed on the target image based on the target image feature to obtain an image processing result.

[0137] Wherein, the content of the first loop end condition and the second loop end condition has been described above, and will not be described herein again.

[0138] The image processing method is described from the perspective of method steps above, and the image processing method of the embodiments of the present application is further described below in combination with Figures 3 to 6 ​It can be concluded that the image processing model in the embodiments of the present application includes a DST block, and the DST block includes a DSSA block. On the one hand, the DSSA block can perform global feature extraction on the first image feature to obtain a global feature, so as to obtain a global receptive field. On the other hand, the DSSA block can perform local feature extraction on the first image feature to obtain a plurality of local features, so as to obtain a local receptive field. Moreover, the global feature extraction and the plurality of local feature extractions are parallel, which can effectively fuse the global information and the local information.

[0139] Please refer to Figure 8 , Figure 8 is a schematic diagram of a receptive field provided by the embodiments of the present application. After the input feature (i.e., the first image feature mentioned above) is input into the DSSA block, the DSSA block can determine the output feature (i.e., the first spliced feature mentioned above). Figure 8 The size of the region to which the output feature is mapped to the input feature is shown. On the one hand, the output feature can be mapped to a local region of the input feature, which indicates that the DSSA block can obtain a local receptive field. On the other hand, the output feature can be mapped to the entire input feature, which indicates that the DSSA block can obtain a global receptive field. In addition, the embodiments of the present application also update the output feature based on the input feature, and input the updated output feature into the DSFFN block, so as to determine the third image feature by using the DSFFN block.

[0140] Please refer to Figure 9 , Figure 9 is a schematic diagram of the relationship between the prediction performance and the computational amount provided by the embodiments of the present application. Wherein, Figure 9 includes the linear relationship between the prediction performance and the computational amount corresponding to each of the model 1 to the model 6. The unit of the computational amount is Billions (B), and the unit of the prediction performance is percentage (%). In the embodiments of the present application, the model 1 is Figure 6 the remaining part of the image processing model shown in the image processing model except for the image processing network layer, which is mainly used for extracting the target image feature of the image, and can be called a backbone network. The model 2 to the model 6 are respectively five other backbone networks except for the model 1. From Figure 9 It can be seen that the computational amount of the model 1 is relatively small, but the prediction performance is high, so it can be concluded that, Figure 6 the accuracy and the operation speed of the image processing model shown in the image processing model are high.

[0141] Please refer to Table 1 below. Table 1 shows the prediction performance and the computational amount of the model 1 to the model 6. Wherein, Figure 9The relationship between the prediction performance and the calculation amount shown is obtained according to the data shown in Table 1 (but not limited to the data shown in Table 1). In the embodiments of the present application, for any one of the models 1 to 6 (denoted as model A), the number of model A is at least one. The form of model A-B is used to represent the Bth model A, and B is a positive integer.

[0142] Table 1

[0143] Model Computational load Prediction performance Model 1-1 2.5B 81.8％ Model 2-1 2.9B 81.7％ Model 3-1 3.8B 79.8％ Model 4-1 4.1B 76.2％ Model 1-2 3.8B 82.6％ Model 5-1 4.0B 82.0％ Model 6-1 4.5B 81.3％ Model 3-1 6.7B 81.2％ Model 4-2 7.9B 77.8％ Model 1-3 6.9B 83.9％ Model 5-2 6.9B 83.2％ Model 2-2 8.6B 83.2％ Model 6-2 8.7B 83.0％ Model 4-3 9.9B 83.6％ Model 2-3 15.1B 83.7％ Model 6-3 15.4B 83.3％

[0144] As can be seen from Table 1, the calculation amount of model 1 is relatively small, but the prediction performance is relatively high, so Figure 6 The accuracy and operation speed of the image processing model shown are high.

[0145] Please refer to Table 2, which shows information corresponding to different image processing models. Among them, the image processing model is obtained by connecting different image processing network layers (denoted as network layer 1 to 6 respectively) in series after the backbone network, and the backbone network includes the backbone network corresponding to model 1, model 2, model 4 and model 6.

[0146] The embodiments of the present application can obtain a verification data set and a test data set, and divide the verification data set into a single-scale data set and a multi-scale data set. The prediction performance of each image processing model is tested using the single-scale data set, and a single-scale score (unit: %) is obtained. Similarly, the prediction performance of each image processing model is tested using the multi-scale data set, and a multi-scale score (unit: %) is obtained. The prediction performance of each image processing model is tested using the test data set, and the corresponding score (unit: %) is obtained. In addition, the embodiments of the present application also test the parameter amount and the calculation amount of each image processing model, wherein the parameter amount is in units of megabytes (MByte, MB), and the calculation amount is in units of billion (Giga, G).

[0147] Table 2

[0148]

[0149] As can be seen from Table 2, the prediction performance of the two image processing models obtained by connecting network layer 5 and network layer 6 in series after model 1 is good, which can improve the accuracy of the image processing result. And the parameter amount and the calculation amount of these two models are relatively small, which can improve the image processing efficiency.

[0150] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the target image involved in the present application is obtained under sufficient authorization.

[0151] Figure 10 As shown in FIG. 1, the image processing device provided by the embodiment of the present application includes an acquisition module 1101, a downsampling processing module 1102, a global feature extraction module 1103, a local feature extraction module 1104 and an image processing module 1105. Figure 10

[0152] The acquisition module 1101 is configured to acquire a target image and an image processing model.

[0153] The downsampling processing module 1102 is configured to input the target image into the image processing model, perform downsampling processing on the target image by the image processing model, and obtain a first image feature of the target image.

[0154] The global feature extraction module 1103 is configured to perform global feature extraction on the first image feature, and obtain a global feature of the target image.

[0155] The local feature extraction module 1104 is configured to perform local feature extraction on the first image feature, and obtain a plurality of local features of the target image.

[0156] The image processing module 1105 is configured to perform image processing on the target image based on the global feature and the plurality of local features, and obtain an image processing result of the target image.

[0157] In a possible implementation, the global feature extraction module 1103 is configured to perform downsampling processing on the first image feature, and obtain a second image feature of the target image; and perform global feature extraction on the second image feature, and obtain the global feature of the target image.

[0158] In a possible implementation, the local feature extraction module 1104 is configured to perform block processing on the first image feature, and obtain a plurality of feature blocks; and for any feature block, perform feature extraction on the feature block, and obtain a local feature of the target image.

[0159] In a possible implementation, the image processing module 1105 is configured to determine a third image feature of the target image based on the global feature and the plurality of local features; determine a target image feature of the target image based on the third image feature; and perform image processing on the target image based on the target image feature, and obtain the image processing result of the target image.

[0160] ​In a possible implementation, the image processing module 1105 is configured to splice the global feature and the plurality of local features to obtain first spliced features of the target image; perform convolution processing on the first spliced features at at least two scales to obtain features corresponding to the respective scales; splice the features corresponding to the respective scales to obtain second spliced features of the target image; and determine third image features of the target image based on the second spliced features.

[0161] In a possible implementation, the image processing module 1105 is configured to, in response to the first loop end condition being met and the second loop end condition being met, take the third image features as the target image features; in response to the second loop end condition not being met, take the third image features as the first image features of the next loop, and start a loop from performing global feature extraction on the first image features to obtain the global features of the target image, until the first loop end condition is met and the second loop end condition is met, to obtain the target image features; and in response to the first loop end condition not being met but the second loop end condition being met, take the third image features as the target image of the next loop, and start a loop from performing down-sampling processing on the target image to obtain the first image features of the target image, until the first loop end condition is met and the second loop end condition is met, to obtain the target image features.

[0162] In a possible implementation, the image processing model includes at least one feature extraction network layer in series, and each feature extraction network layer includes at least one feature extraction network block in series; the first loop end condition is met when the current loop is a loop in a last feature extraction network layer of the at least one feature extraction network layer in series; and the second loop end condition is met when the current loop is a loop in a last feature extraction network block of the at least one feature extraction network block in series included in any feature extraction network layer.

[0163] The apparatus described above first performs down-sampling processing on the target image to obtain the first image features, and the down-sampling processing reduces the computational load of subsequent processing. Then, the global feature extraction is performed on the first image features to obtain the global features, and the local feature extraction is performed on the first image features to obtain the plurality of local features, which realizes the extraction of the double-scale image features, so that the accuracy of image processing based on the global features and the plurality of local features can be improved, and the efficiency of image processing is improved due to the low computational load.

[0164] It should be understood that the above Figure 10The apparatus provided in the embodiments is only taken as an example for the division of the functional modules, and in actual application, the above-mentioned functions can be completed by different functional modules according to the needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method provided in the above-mentioned embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0165] Figure 11 A structure block diagram of a terminal device 1200 provided in an example embodiment of the present application is shown. The terminal device 1200 includes a processor 1201 and a memory 1202.

[0166] The processor 1201 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 1201 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed on the display screen. In some embodiments, the processor 1201 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0167] The memory 1202 can include one or more computer-readable storage media, which can be non-transitory. The memory 1202 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one computer program for being executed by the processor 1201 to implement the image processing method provided in the method embodiments of the present application.

[0168] In some embodiments, the terminal device 1200 can further optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1203 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1204, a display screen 1205, a camera component 1206, an audio circuit 1207 and a power supply 1208.

[0169] The peripheral device interface 1203 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202 and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202 and the peripheral device interface 1203 can be implemented on a separate chip or circuit board, and the present embodiments are not limited thereto.

[0170] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 can also include NFC (Near Field Communication) related circuitry, which is not limited in the present application.

[0171] The display screen 1205 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 is further configured to capture touch signals on or above the surface of the display screen 1205. The touch signals can be input to the processor 1201 as control signals for processing. In this case, the display screen 1205 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1205 can be one, arranged on the front panel of the terminal device 1200; in other embodiments, the display screen 1205 can be at least two, arranged on different surfaces of the terminal device 1200 or in a folding design; in other embodiments, the display screen 1205 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal device 1200. Even, the display screen 1205 can also be arranged in an irregular shape, i.e. a special-shaped screen. The display screen 1205 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0172] The camera assembly 1206 is configured to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, the rear camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1206 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0173] The audio circuit 1207 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 1201 for processing, or input to the radio frequency circuit 1204 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal device 1200. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 1207 can also include a headphone jack.

[0174] The power supply 1208 is used to supply power to each component in the terminal device 1200. The power supply 1208 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1208 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0175] In some embodiments, the terminal device 1200 further includes one or more sensors 1209. The one or more sensors 1209 include, but are not limited to, an acceleration sensor 1211, a gyroscope sensor 1212, a pressure sensor 1213, an optical sensor 1214, and a proximity sensor 1215.

[0176] The acceleration sensor 1211 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the terminal device 1200. For example, the acceleration sensor 1211 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 1201 can control the display screen 1205 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1211. The acceleration sensor 1211 can also be used for game or user motion data collection.

[0177] The gyroscope sensor 1212 can detect the body direction and rotation angle of the terminal device 1200, and the gyroscope sensor 1212 can collect 3D actions of the user on the terminal device 1200 in cooperation with the acceleration sensor 1211. The processor 1201 can realize the following functions according to the data collected by the gyroscope sensor 1212: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0178] The pressure sensor 1213 can be disposed on the side bezel of the terminal device 1200 and / or on the lower layer of the display screen 1205. When the pressure sensor 1213 is disposed on the side bezel of the terminal device 1200, it can detect the user's grip signal on the terminal device 1200, and the processor 1201 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1213. When the pressure sensor 1213 is disposed on the lower layer of the display screen 1205, the processor 1201 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1205. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0179] Optical sensor 1214 is used to collect ambient light intensity. In one embodiment, processor 1201 can control the display brightness of display screen 1205 based on the ambient light intensity collected by optical sensor 1214. Specifically, when the ambient light intensity is high, the display brightness of display screen 1205 is increased; when the ambient light intensity is low, the display brightness of display screen 1205 is decreased. In another embodiment, processor 1201 can also dynamically adjust the shooting parameters of camera assembly 1206 based on the ambient light intensity collected by optical sensor 1214.

[0180] The proximity sensor 1215, also known as a distance sensor, is typically installed on the front panel of the terminal device 1200. The proximity sensor 1215 is used to detect the distance between the user and the front of the terminal device 1200. In one embodiment, when the proximity sensor 1215 detects that the distance between the user and the front of the terminal device 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from a screen-on state to a screen-off state; when the proximity sensor 1215 detects that the distance between the user and the front of the terminal device 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from a screen-off state to a screen-on state.

[0181] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on the terminal device 1200, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0182] Figure 12A structural diagram of a server provided in the embodiments of the present application is shown in FIG. 13. The server 1300 can have great differences due to different configurations or performances, and can include one or more processors 1301 and one or more memories 1302. The one or more memories 1302 store at least one computer program, which is loaded and executed by the one or more processors 1301 to implement the image processing method provided in any of the above method embodiments. The processor 1301 is, for example, a CPU. Of course, the server 1300 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for implementing device functions, and the like, so as to perform input and output. The server 1300 can also include other components for implementing device functions, which are not described herein.

[0183] In exemplary embodiments, a computer readable storage medium is also provided, which stores at least one computer program. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above image processing methods.

[0184] Optionally, the above computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0185] In exemplary embodiments, a computer program or computer program product is also provided, which stores at least one computer program. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above image processing methods.

[0186] It should be understood that "multiple" referred to herein means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0187] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0188] The above description is only exemplary embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, and the like made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a target image and an image processing model; inputting the target image into the image processing model, performing down-sampling processing on the target image by the image processing model to obtain a first image feature of the target image; performing global feature extraction on the first image feature to obtain a global feature of the target image; performing local feature extraction on the first image feature to obtain a plurality of local features of the target image; splicing the global feature and the plurality of local features to obtain a first spliced feature of the target image; performing convolution processing on the first spliced feature in at least two scales to obtain features corresponding to respective scales; splicing the features corresponding to the respective scales to obtain a second spliced feature of the target image; determining a third image feature of the target image based on the second spliced feature; determining a target image feature of the target image based on the third image feature; performing image processing on the target image based on the target image feature to obtain an image processing result of the target image.

2. The method of claim 1, wherein, The global feature extraction on the first image feature to obtain the global feature of the target image comprises: performing down-sampling processing on the first image feature to obtain a second image feature of the target image; performing global feature extraction on the second image feature to obtain the global feature of the target image.

3. The method of claim 1, wherein, The local feature extraction on the first image feature to obtain the plurality of local features of the target image comprises: performing block processing on the first image feature to obtain a plurality of feature blocks; for any one feature block, performing feature extraction on the any one feature block to obtain one local feature of the target image.

4. The method of claim 1, wherein, The determination of the target image feature of the target image based on the third image feature comprises: in response to satisfying a first loop end condition and satisfying a second loop end condition, taking the third image feature as the target image feature; in response to not satisfying the second loop end condition, taking the third image feature as a first image feature of a next loop, and starting to perform a loop from the global feature extraction on the first image feature to obtain the global feature of the target image until the first loop end condition is satisfied and the second loop end condition is satisfied to obtain the target image feature; in response to not satisfying the first loop end condition but satisfying the second loop end condition, taking the third image feature as a target image of a next loop, and starting to perform a loop from the down-sampling processing on the target image to obtain the first image feature of the target image until the first loop end condition is satisfied and the second loop end condition is satisfied to obtain the target image feature.

5. The method of claim 4, wherein, The image processing model comprises at least one feature extraction network layer in series, and each feature extraction network layer comprises at least one feature extraction network block in series; the first loop end condition is that the current loop is a loop in a last feature extraction network layer of the at least one feature extraction network layer in series. The second loop end condition is satisfied when the current loop is the last loop in at least one serial feature extraction network block included in the any feature extraction network layer.

6. An image processing apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire a target image and an image processing model; A down-sampling processing module is configured to input the target image into the image processing model, perform down-sampling processing on the target image by the image processing model, and obtain a first image feature of the target image; A global feature extraction module is configured to perform global feature extraction on the first image feature, and obtain a global feature of the target image; A local feature extraction module is configured to perform local feature extraction on the first image feature, and obtain a plurality of local features of the target image; An image processing module is configured to splice the global feature and the plurality of local features to obtain a first spliced feature of the target image, perform convolution processing on the first spliced feature in at least two scales to obtain features corresponding to the respective scales, splice the features corresponding to the respective scales to obtain a second spliced feature of the target image, determine a third image feature of the target image based on the second spliced feature, determine a target image feature of the target image based on the third image feature, and perform image processing on the target image based on the target image feature to obtain an image processing result of the target image.

7. The apparatus of claim 6, wherein, The global feature extraction module is configured to perform down-sampling processing on the first image feature to obtain a second image feature of the target image, and perform global feature extraction on the second image feature to obtain a global feature of the target image.

8. The apparatus of claim 6, wherein, The local feature extraction module is configured to perform block processing on the first image feature to obtain a plurality of feature blocks, and perform feature extraction on any feature block to obtain a local feature of the target image.

9. The apparatus of claim 6, wherein, The image processing module is configured to: In response to satisfying a first loop end condition and a second loop end condition, the third image feature is taken as the target image feature; In response to not satisfying the second loop end condition, the third image feature is taken as a first image feature of the next loop, and a loop is executed from the global feature extraction on the first image feature to obtain the global feature of the target image until the first loop end condition and the second loop end condition are satisfied to obtain the target image feature; In response to not satisfying the first loop end condition but satisfying the second loop end condition, the third image feature is taken as a target image of the next loop, and a loop is executed from the down-sampling processing on the target image to obtain the first image feature of the target image until the first loop end condition and the second loop end condition are satisfied to obtain the target image feature.

10. The apparatus of claim 9, wherein, The image processing model comprises at least one serial feature extraction network layer, and any feature extraction network layer comprises at least one serial feature extraction network block. The first loop end condition is met when the current loop is a loop in a last feature extraction network layer of the at least one series of feature extraction network layers; The second loop end condition is met when the current loop is a loop in a last feature extraction network block of the at least one series of feature extraction network blocks included in the any feature extraction network layer.

11. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the image processing method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the image processing method of any one of claims 1 to 5.

13. A computer program product, characterised in that, The computer program product stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the image processing method of any one of claims 1 to 5.

Images