Image feature processing method and device of neural network, equipment and medium
By combining fully convolutional networks and spatial pyramid pooling layers, the requirement of fixed image size for the Transformer model is solved, enabling accurate processing of images of arbitrary sizes and improving the generalization and processing speed of the neural network.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU RIPPLEINFO TECH CO LTD
- Filing Date
- 2021-10-15
- Publication Date
- 2026-04-10
AI Technical Summary
Existing Transformer models require fixed image dimensions when processing image features, which leads to changes in the original image data and details, resulting in inaccurate output results.
This paper employs a combination of fully convolutional networks, spatial pyramid pooling layers, and Transformers to segment images into blocks using image segmentation methods. The neural network is trained using a loss function and backpropagation algorithm, supporting inputs of arbitrary image sizes. Furthermore, the spatial pyramid pooling layers convert features into fixed dimensions, enabling accurate processing of image features.
It supports image input of any size, preserves original data and detailed information, expands the generalization ability of neural networks, and improves the accuracy and speed of image feature processing.
Smart Images
Figure CN115994913B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of image processing, and in particular to an image feature processing method and device of a neural network, an image feature processing apparatus and medium. BACKGROUND
[0002] With the development of the field of artificial intelligence, neural networks have emerged. As a deep neural network mainly based on self-attention mechanism, the Transformer is widely used in the field of natural language processing. The Transformer can be extended to computer vision tasks. Compared with other network types (such as convolutional networks and recurrent networks), the model based on the Transformer shows competitive and even better performance on various visual benchmarks, and achieves good results.
[0003] However, when using the Transformer, since the Transformer needs to input and process image features of a fixed dimension, the input size of the image and the features of the image must be fixed during processing, which may cause changes in the data and details of the original image, and in turn cause the output result to be inaccurate when processed by the Transformer. SUMMARY
[0004] Therefore, it is necessary to provide an image feature processing method, device, apparatus and medium of a neural network capable of supporting an input size of any image in view of the above technical problems.
[0005] An image feature processing method of a neural network, the method comprising:
[0006] inputting an original image into a fully convolutional network of a pre-trained neural network to output a first feature image, the pre-trained neural network comprising: inputting a training image into the neural network, sequentially processing the training image through a fully convolutional network, a spatial pyramid pooling layer and a Transformer of the neural network to output a processing result of the training image, the training image comprising at least one image; calculating a loss value of the processing result of the training image through a loss function; obtaining the pre-trained neural network in a case where the loss value is less than or equal to a preset loss threshold value;
[0007] cutting the first feature image through an image segmentation method to generate a first feature image sequence;
[0008] inputting the first feature image sequence into the spatial pyramid pooling layer of the pre-trained neural network to perform pooling processing through the spatial pyramid pooling layer to obtain a first feature sequence of a fixed dimension;
[0009] The first feature sequence is input into a Transformer of the pre-trained neural network, and a processing result is output by the Transformer.
[0010] In one of the embodiments, the training image is input into the neural network, sequentially processed by a full convolution network, a spatial pyramid pooling layer and a Transformer of the neural network, and a processing result of the training image is output, including:
[0011] The second image sequence of the training image is input into the full convolution network of the neural network, and a second feature image sequence is output.
[0012] The second feature image sequence is input into the spatial pyramid pooling layer of the neural network, and a pooling processing is performed by the spatial pyramid pooling layer to obtain a second feature sequence of a fixed dimension.
[0013] The second feature sequence of the fixed dimension is input into the Transformer of the neural network, and the processing result of the training image is output.
[0014] In one of the embodiments, the second image sequence of the training image is input into the full convolution network of the neural network, and the following is further included:
[0015] The training image is segmented into a plurality of image blocks by an image segmentation method, and the plurality of image blocks are taken as the second image sequence of the training image.
[0016] In one of the embodiments, after the loss value of the processing result of the training image is calculated by the loss function, the following is further included:
[0017] In a case where the loss value is greater than a preset loss threshold, the weights of the neural network are updated by a back propagation algorithm;
[0018] After the weights of the neural network are updated, the neural network with the updated weights is used to perform the training operation again until the loss value is less than or equal to the preset loss threshold.
[0019] In one of the embodiments, the pooling processing includes:
[0020] A first scale pooling processing is performed by the spatial pyramid pooling layer, a first average value of the first scale is calculated, and the first average value is taken as a first scale pooling feature;
[0021] A second scale pooling processing is performed by the spatial pyramid pooling layer, a second average value of the second scale is calculated, and the second average value is taken as a second scale pooling feature;
[0022] The first scale pooling feature and the second scale pooling feature are used to calculate the dimension of the feature sequence.
[0023] In one of the embodiments, the second image sequence of the training image is input into the full convolution network of the neural network, comprising:
[0024] The image blocks in the second image sequence are sequentially input into the first four layers of the convolution network in the residual network, the features of the image blocks are extracted by the residual network, and the feature image blocks are output;
[0025] The feature image blocks are combined to form an image feature sequence.
[0026] In one of the embodiments, before the plurality of image blocks are taken as the second image sequence of the training image, the training image is segmented into a plurality of image blocks by an image segmentation method, further comprising:
[0027] The training images with the same size are batched according to a preset sample number for training.
[0028] In one of the embodiments, after the processing result is output by the Transformer, further comprising:
[0029] An image processing task is performed by the processing result, and the image processing task includes image classification, target detection, and semantic segmentation.
[0030] An image feature processing device of a neural network, comprising:
[0031] A neural network training module is configured to perform a training operation, the training operation comprising: inputting a training image into a neural network, sequentially processing the training image through a full convolution network, a spatial pyramid pooling layer, and a Transformer of the neural network, thereby outputting a processing result of the training image, the training image comprising at least one image; calculating a loss value of the processing result of the training image by a loss function; and obtaining a pre-trained neural network when the loss value is less than or equal to a preset loss threshold.
[0032] A feature image output module is configured to input an original image into the full convolution network of the pre-trained neural network, and output a first feature image.
[0033] An image segmentation module is configured to cut the first feature image by an image segmentation method, and generate a first feature image sequence.
[0034] A pooling module is configured to pass the first feature image sequence through the spatial pyramid pooling layer of the pre-trained neural network, and perform pooling processing by the spatial pyramid pooling layer to obtain a first feature sequence with a fixed dimension.
[0035] A Transformer processing module is configured to input the first feature sequence into the Transformer of the pre-trained neural network, and output a processing result by the Transformer.
[0036] A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of the method described above when executing the computer program.
[0037] A computer readable storage medium having stored thereon a computer program, the computer program implementing the steps of the method described above when executed by a processor.
[0038] A computer program product comprising a computer program, the computer program implementing the steps of the method described above when executed by a processor.
[0039] The image feature processing method, device, computer device and storage medium of the neural network described above, by introducing a full convolutional network to provide features for the Transformer, can support inputting images of any size into the neural network, without the need for adjusting the size of the original image and other operations, preserving the original image data and image detail information. In addition, by introducing a spatial pyramid pooling, the output features of the full convolutional network of any size are converted to a fixed dimension as the input of the image Transformer, so that the input image features do not need to be fixed in size, expanding the generalization of the neural network, canceling the constraints of the neural network on the feature size, and also being able to converge the features, and finally through the Transformer output, the output result can be more accurate.
[0040] On the other hand, by introducing a loss function, and further verifying whether the neural network converges through the loss function, a neural network with better training can be obtained, thereby improving the accuracy of image feature processing, and by introducing a backpropagation algorithm, in the case of a larger loss value, the weights of the neural network are updated to reduce the case of a too large loss value caused by the weights, further improving the accuracy of the neural network training.
[0041] On the other hand, the training image is first segmented into image blocks using an image segmentation method, and then feature extraction is performed, which can improve the accuracy of training the neural network, and in the image feature processing method, the entire original image is loaded into the GPU at one time for convolution operation, reducing the data scheduling between the GPU and the CPU, and further improving the speed of image feature processing. BRIEF DESCRIPTION OF DRAWINGS
[0042] Figure 1 An application environment schematic diagram of the image feature processing method of the neural network in one embodiment;
[0043] Figure 2 A flowchart schematic diagram of the image feature processing method of the neural network in one embodiment;
[0044] Figure 3 Fig. 2 is a schematic diagram of a FCN network structure in a neural network according to an embodiment;
[0045] Figure 4 Fig. 3 is a schematic diagram of a spatial pyramid pooling layer structure in a neural network according to an embodiment;
[0046] Figure 5 Fig. 4 is a schematic diagram of a Transformer structure in a neural network according to an embodiment;
[0047] Figure 6 Fig. 5 is a schematic diagram of a flow of steps of a training operation according to an embodiment;
[0048] Figure 7 Fig. 6 is a schematic diagram of a flow of steps of S602 according to an embodiment;
[0049] Figure 8 Fig. 7 is a schematic diagram of a flow of steps of a pooling processing according to an embodiment;
[0050] Figure 9 Fig. 8 is a schematic diagram of a training stage and a processing stage of an image feature processing method of a neural network according to another embodiment;
[0051] Figure 10 Fig. 9 is a schematic diagram of a structure of an image feature processing device of a neural network according to an embodiment;
[0052] Figure 11 Fig. 10 is a schematic diagram of an internal structure of a computer device according to an embodiment. DETAILED DESCRIPTION
[0053] In order to make the objects, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and not intended to limit the present disclosure.
[0054] It should be noted that the terms "first", "second" and the like in the description and claims of the present document and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present document described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or apparatus that comprises a list of steps or units does not necessarily limit those steps or units to the clearly listed ones, but can include other steps or units not clearly listed or inherent to such processes, methods, products or apparatus.
[0055] The image feature processing method of the neural network provided by the present disclosure can be applied to the application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed or stored by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The terminal 102 inputs the original image into the fully convolutional network of the neural network stored in the data storage system pre-trained by the server 104, and outputs the first feature image of the original image through the fully convolutional network. The first feature image output by the server 104 is cut through an image segmentation method, and a first feature image sequence is generated after cutting. The first feature image sequence is input into the spatial pyramid pooling layer of the neural network in the server 104, and the spatial pyramid is used for pooling processing to obtain a fixed first feature sequence. The first feature sequence is input into the Transformer of the neural network in the server 104, and the processing result of the first feature sequence is output through the Transformer. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers. It can be understood that the method can also be applied to a terminal or a server alone to implement.
[0056] In one embodiment, as shown in Figure 2 An image feature processing method of a neural network is provided. Taking the server in Figure 1 as an example, the method includes the following steps:
[0057] S202, the terminal 102 sends an original image to the server 104, and the server 104 inputs the original image into a fully convolutional network of a pre-trained neural network and outputs a first feature image.
[0058] The original image can be an image required to perform an image processing task, and can include a human face image, an animal image, a background image, etc. The pre-trained neural network can be a neural network trained to process the original image. It can include three parts: a fully convolutional network, a spatial pyramid pooling layer and a Transformer. The first feature image is usually a feature image of the original image output by the fully convolutional network. The fully convolutional network can usually be a FCN (Fully Convolutional Network) network. As shown in Figure 3 , it is a FCN network structure diagram.
[0059] Specifically, the original image is input into a fully convolutional network of a pre-trained neural network, parameters of the fully convolutional network are shared, feature extraction is performed through the fully convolutional network, and a first feature image of the original image is output. The original image can be an image of any size and any dimension.
[0060] In S204, the first feature image is cut by an image segmentation method to generate a first feature image sequence.
[0061] The image segmentation method can be a method of segmenting an image and cutting the image into image blocks.
[0062] Specifically, the first feature image is cut by an image segmentation method. The image segmentation method can include threshold segmentation, region segmentation, edge segmentation, etc. In the embodiments of the present application, there is no limitation as long as the first feature image can be cut into feature image blocks. After cutting, a plurality of feature image blocks are obtained. The plurality of feature image blocks are used as the first feature image sequence. The plurality of feature image blocks in the first feature image sequence can be arranged in any manner, which is not limited in the embodiments.
[0063] In S206, the first feature image sequence is input into a spatial pyramid pooling layer of a pre-trained neural network, and a first feature sequence of a fixed dimension is obtained by performing pooling processing through the spatial pyramid pooling layer.
[0064] The spatial pyramid pooling layer can be a spatial pyramid pooling (SPP) as shown in Figure 4 The pooling processing can be a method of processing through the spatial pyramid pooling layer. The fixed dimension can be determined according to the scale of the pooling processing.
[0065] Specifically, the feature image blocks in the first feature image sequence are sequentially input into the spatial pyramid pooling layer, the pooling processing is performed through the spatial pyramid pooling layer, the feature blocks are transformed into the fixed dimension, and the fixed dimension feature blocks are combined, that is, the first feature sequence of the fixed dimension is obtained.
[0066] In S208, the first feature sequence is input into a Transformer of a pre-trained neural network, and a processing result is output through the Transformer.
[0067] The Transformer can be a deep neural network mainly based on a self-attention mechanism as shown in Figure 5 The Transformer is a structure diagram as shown in
[0068] Specifically, the first feature sequence is input into the Transformer, and the first feature sequence is processed by the Transformer to output a processing result.
[0069] In the image feature processing method of the neural network, the full convolution network is introduced to provide features for the Transformer, so that an image of any size can be input into the neural network without the need for adjusting the size of the original image or other operations, and the original data and details of the image are preserved. In addition, the spatial pyramid pooling is introduced to convert the output features of the full convolution network of any size into a fixed dimension as the input of the image Transformer, so that the input image features do not need to be fixed in size, the generalization of the neural network is expanded, the neural network is no longer bound by the size of the features, and the features can also be aggregated. Moreover, the output result can be more accurate when finally output by the Transformer.
[0070] In one embodiment, as shown in Figure 6 The training operation of the pre-trained neural network includes:
[0071] S602, input a training image into the neural network, and sequentially process the training image through the full convolution network, the spatial pyramid pooling layer and the Transformer of the neural network to output a processing result of the training image, the training image including at least one image.
[0072] Specifically, the specific operation of the training image processed by the full convolution network, the spatial pyramid pooling layer and the Transformer of the neural network can refer to the method steps of the processing of the original image in the above-mentioned embodiments, which will not be described in detail here.
[0073] S604, calculate the loss value of the processing result of the training image by a loss function.
[0074] Specifically, the loss function can be Cross Entropy Loss, i.e. the cross-entropy loss function, which can be:
[0075]
[0076] wherein, is the number of classes of the sample, denotes the sign function, which is 0 or 1, c is the sample class, and if the true class of the sample i is equal to c, 1 is taken, otherwise 0 is taken, is the probability that the observed sample i belongs to the class c,
[0077] N is the number of training samples, is the loss value of the i-th sample in the training samples, and L is the average loss value of the training samples.
[0078] S606, when the loss value is less than or equal to a preset loss threshold, a pre-trained neural network is obtained.
[0079] Specifically, the purpose of training a neural network model is to reduce the loss value. A relatively small value, i.e., a preset loss threshold, can be set. During each training session, the loss value is calculated. When the loss value is less than the preset loss threshold, the model can be considered to have converged. Training can then be stopped. That is, when the loss value is less than the preset loss threshold, it means that the neural network model has converged, training is stopped, and the neural network is obtained. This neural network can be considered a pre-trained neural network.
[0080] In this embodiment, a loss function is introduced to verify whether the neural network has converged, thereby obtaining a neural network with better training performance and improving the accuracy of image feature processing.
[0081] In one embodiment, such as Figure 7 As shown, in step S602, the training image is input into the neural network and processed sequentially through the fully convolutional network, spatial pyramid pooling layer, and Transformer of the neural network, thereby outputting the processing result of the training image, including:
[0082] S702, the training image is segmented into multiple image blocks using an image segmentation method, and the multiple image blocks are used as the second image sequence of the training image, wherein the training image includes at least one image.
[0083] S704 inputs the second image sequence of the training images into the fully convolutional network of the neural network and outputs the second feature image sequence.
[0084] S706, The second feature image sequence is input into the spatial pyramid pooling layer of the neural network, and pooling is performed through the spatial pyramid pooling layer to obtain a second feature sequence with fixed dimensions;
[0085] S708, the fixed-dimensional second feature sequence is input into the Transformer in the neural network, and the processing result of the training image is output.
[0086] Specifically, the specific operations of the training images in the neural network are described in the above embodiments, which are the processing steps of the original images in the pre-trained neural network. They will not be elaborated in detail here.
[0087] In the embodiment, the training image is segmented into image blocks by the image segmentation method, and then feature extraction is performed, mainly to maintain consistency with other image processing methods, facilitate module replacement in the training framework, and in the image feature processing method, the entire original image is loaded into the GPU at one time for convolution operation, reducing data scheduling between the GPU and the CPU, and thereby improving the speed of image feature processing.
[0088] In one embodiment, the loss value of the processing result of the training image is calculated by the loss function, and then the method further comprises:
[0089] In the case where the loss value is greater than the preset loss threshold, the weights of the neural network are updated by a back propagation algorithm;
[0090] The weights of the neural network are updated, and the training operation is performed again using the neural network with updated weights until the loss value is less than or equal to the preset loss threshold.
[0091] The back propagation algorithm can be a learning algorithm suitable for a multi-layer neural network. The back propagation algorithm mainly comprises two links (excitation propagation and weight update) which are repeatedly iterated until the response of the network to the input reaches a predetermined target range.
[0092] Specifically, in the case where the loss value is greater than the preset loss threshold, it is proved that the neural network training is not good, and the weights of the neural network are updated by the back propagation algorithm to reduce the loss value. After updating the weights, the training operation mentioned in the above embodiment is performed again until the loss value is less than the preset loss threshold.
[0093] In the embodiment, by introducing the back propagation algorithm, in the case where the loss value is large, the weights of the neural network are updated to reduce the case where the loss value is too large due to the weights, and the accuracy of the neural network training is further improved.
[0094] In one embodiment, as shown in Figure 8 The pooling processing comprises:
[0095] S802, performing first scale pooling processing by the spatial pyramid pooling layer, calculating a first average value of the first scale, and taking the first average value as a first scale pooling feature;
[0096] S804, performing second scale pooling processing by the spatial pyramid pooling layer, calculating a second average value of the second scale, and taking the second average value as a second scale pooling feature;
[0097] S806, calculating the dimension of the feature sequence using the first scale pooling feature and the second scale pooling feature.
[0098] Specifically, when the first feature image sequence is subjected to the pooling processing, the image blocks in the first feature image sequence are sequentially input to the spatial pyramid pooling layer, each feature image block is subjected to the first scale pooling processing by the spatial pyramid pooling layer, and a first average value of the first scale is calculated, the first average value is taken as a first scale pooling feature, a feature image block of the first scale pooling feature is obtained, thereby forming a feature sequence of the first scale pooling feature, then the image blocks in the first feature image sequence are sequentially input to the spatial pyramid pooling layer, each feature image block is subjected to the second scale pooling processing by the spatial pyramid pooling layer, and a second average value of the second scale is calculated, the second average value is taken as a second scale pooling feature, a feature image block of the second scale pooling feature is obtained, thereby forming a feature sequence of the second scale pooling feature, after the feature sequence of the first scale pooling feature and the feature sequence of the second scale pooling feature are calculated, one feature is extracted from each feature image block in the feature sequence of the first scale pooling feature and the feature sequence of the second scale pooling feature, and finally a first feature sequence with a fixed dimension is obtained.
[0099] In some embodiments, the image blocks in the first feature image sequence are sequentially input to the spatial pyramid pooling layer, each feature image block is subjected to the first scale pooling processing by the spatial pyramid pooling layer, the first scale can be 16, and the first average value of the first scale is 4, which is taken as the first scale pooling feature, thereby obtaining a feature sequence of 4*4*256, wherein 256 is the image pixel, the second scale can be 1, and the second average value of the second scale is also 1, thereby obtaining a feature sequence of 1*1*256, the feature sequence of 4*4*256 and the feature sequence of 1*1*256 are combined, thereby obtaining a feature sequence of 1*4352, which is converted into a first feature sequence with a fixed dimension of 17*256.
[0100] Specifically, when the second feature image sequence is subjected to the pooling processing, the specific operation can be referred to the operation when the first feature image sequence is processed, which is not described in detail herein.
[0101] It should be noted that the first scale and the second scale can be selected and set by the person skilled in the art according to the specific situation, which is not limited in the present disclosure.
[0102] In the present embodiment, the main role of the pooling processing is to make the feature information value smoother, and at the same time reduce the amount of feature information, and the person skilled in the art can also select three scales, four scales and the like for the pooling processing according to the actual situation, and the preferred scheme in the present disclosure is to perform the pooling processing by two scales.
[0103] In an embodiment, the inputting the second image sequence of the training image into a fully convolutional network of the neural network outputs an image feature sequence, including:
[0104] The image blocks in the second image sequence are sequentially inputted into the first four layers of the convolutional network in the residual network, and the features of the image blocks are extracted by the residual network to output a feature image block.
[0105] The feature image blocks are combined to form an image feature sequence.
[0106] The residual network can be a network of ResNet structure, which is a convolutional neural network.
[0107] Specifically, the first four layers of the convolutional network in the residual network ResNet are used as a feature extraction network, and each image block is extracted to a WxHx256 image block feature map through the fully convolutional network. W is the width and H is the depth. The ResNet can be ResNet-18 and ResNet-34, and the degradation problem does not occur in ResNet-18 and ResNet-34. In this embodiment, ResNet-34 is used, which performs better than ResNet-18. Moreover, the residual network is easier to optimize and converges faster.
[0108] In this embodiment, the ResNet structure is very easy to modify and expand. By adjusting the number of channels in the block and the number of stacked blocks, the width and depth of the network can be easily adjusted to obtain networks with different expression capabilities without worrying too much about the "degradation" problem of the network.
[0109] In an embodiment, after the processing result is output by the Transformer, the method further includes:
[0110] The processing result is used to perform an image processing task, which can include image classification, object detection, semantic segmentation, etc.
[0111] In some embodiments, when performing an image classification task, such as a classification task of 20, the processing result output by the Transformer is used to take the first dimension of the processing result, then a fully connected layer is connected, and a CrossEntropy Loss loss function is used for calculation.
[0112] In some embodiments, the training image is divided into a plurality of image blocks by an image segmentation method, and the plurality of image blocks are used as the second image sequence of the training image, and the method further includes:
[0113] The training of the training image with the same size is batched according to a preset sample number.
[0114] Specifically, in the training process, the Batch Size can be set as a preset sample number, and then the training images with the same size are batched and trained according to the preset sample number, so as to improve the training speed.
[0115] wherein the Batch Size is the number of samples selected for one training. Within a certain range, generally speaking, the larger the Batch Size is, the more accurate the determined descending direction is, the smaller the training shock is caused, and the better the training effect is.
[0116] In another embodiment, as shown in Figure 9 a neural network image feature processing method is provided, the method comprising:
[0117] Training stage: the training image is segmented into image blocks, and the image blocks are sequentially input into the full convolution network of the neural network, and then output to the spatial pyramid pooling layer of the neural network after full convolution to convert into fixed-dimension features, and then input into the Transformer of the neural network to obtain a processing result, and calculate a loss value of the processing result, and when the loss value is less than a preset loss threshold, the neural network training is completed.
[0118] Processing stage: using the trained neural network, the original image, which can also be the same as the training image, is input into the full convolution network of the neural network to extract features, and then the extracted features are segmented into feature image blocks, and the feature image blocks are sequentially input into the spatial pyramid pooling layer and the Transformer of the trained neural network to obtain a processing result, and then the image processing task is performed according to the processing result, which can include image classification, target detection, semantic segmentation, etc.
[0119] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the arrow indication, these steps are not necessarily executed in sequence according to the arrow indication. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.
[0120] In one embodiment, as shown in Figure 10As shown, an image feature processing device 1000 of a neural network is provided, comprising: a neural network training module 1020, a feature image output module 1040, an image segmentation module 1060, a pooling module 1080 and a Transformer processing module 1100, wherein:
[0121] The neural network training module 1020 is configured to perform a training operation, and the training operation comprises:
[0122] The training image is input into the neural network, and is sequentially processed by a full convolutional network, a spatial pyramid pooling layer and a Transformer of the neural network, so as to output a processing result of the training image, wherein the training image comprises at least one image;
[0123] The loss value of the processing result of the training image is calculated by a loss function.
[0124] When the loss value is less than or equal to a preset loss threshold, a pre-trained neural network is obtained.
[0125] The feature image output module 1040 is configured to input the original image into the full convolutional network of the pre-trained neural network, and output a first feature image.
[0126] The image segmentation module 1060 is configured to cut the first feature image by an image segmentation method and generate a first feature image sequence.
[0127] The pooling module 1080 is configured to input the first feature image sequence into the spatial pyramid pooling layer of the pre-trained neural network, and perform pooling processing by the spatial pyramid pooling layer, so as to obtain a first feature sequence with a fixed dimension.
[0128] The Transformer processing module 1100 is configured to input the first feature sequence with the fixed dimension into the Transformer of the neural network, and output a processing result by the Transformer.
[0129] In an embodiment of the device, the neural network training module comprises: a training image processing module configured to input a second image sequence of a training image into the full convolutional network of the neural network, and output a second feature image sequence;
[0130] The second feature image sequence is input into the spatial pyramid pooling layer of the neural network, and is processed by the spatial pyramid pooling layer, so as to obtain a second feature sequence with a fixed dimension.
[0131] The second feature sequence with the fixed dimension is input into the Transformer of the neural network, and a processing result of the training image is output.
[0132] In one of the embodiments of the device, the image segmentation module 1060 is further configured to segment the training image into image blocks by an image segmentation method, and combine the image blocks to form a second image sequence of the training image.
[0133] In one of the embodiments of the device, the neural network training module further comprises a weight updating module and a training operation repeating module, configured to update the weight of the neural network by a back propagation algorithm if the loss value is greater than the preset loss threshold value.
[0134] The training operation repeating module is configured to execute the training operation again after updating the weight of the neural network.
[0135] In one of the embodiments of the device, the pooling module comprises a first scale pooling processing module, a second scale pooling processing module, and a feature dimension calculation module.
[0136] The first scale pooling processing module is configured to perform first scale pooling processing by the spatial pyramid pooling layer, calculate a first average value of the first scale, and take the first average value as a first scale pooling feature.
[0137] The second scale pooling processing module is configured to perform second scale pooling processing by the spatial pyramid pooling layer, calculate a second average value of the second scale, and take the second average value as a second scale pooling feature.
[0138] The feature dimension calculation module is configured to calculate the dimension of the feature sequence by using the first scale pooling feature and the second scale pooling feature.
[0139] In one of the embodiments of the device, the feature image output module 1040 comprises a residual network processing module and an image combination module.
[0140] The residual network processing module is configured to input the image blocks in the second image sequence into the first four layers of the convolutional network in the residual network in sequence, extract features of the image blocks by the residual network, and output feature image blocks.
[0141] The image combination module is configured to combine the feature image blocks to form an image feature sequence.
[0142] In one of the embodiments of the device, the device further comprises an image task processing module configured to perform an image processing task by the processing result, wherein the image processing task comprises image classification, target detection, and semantic segmentation.
[0143] The specific implementation of the image feature processing apparatus of the neural network can refer to the description of the implementation of the image feature processing method of the neural network in the foregoing, and will not be described here. Each module in the image feature processing apparatus of the neural network can be implemented by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.
[0144] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 11 The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store training image data and original image data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an image feature processing method of a neural network.
[0145] Those skilled in the art can understand that Figure 11 The structure shown in the figure is only a block diagram of part of the structure related to the present disclosure, and does not constitute a limitation on the computer device to which the present disclosure is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0146] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the steps of the image feature processing method of the neural network in the above-mentioned embodiments.
[0147] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by the processor to implement the steps of the image feature processing method of the neural network in the above-mentioned embodiments.
[0148] In one embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by the processor to implement the steps of the image feature processing method of the neural network in the above-mentioned embodiments.
[0149] It should be noted that the training image data and the original image data involved in the present disclosure are information and data authorized by the user or fully authorized by all parties.
[0150] The specific embodiments of the device, storage medium, computer program product and the like described in the above can refer to the embodiments of the image feature processing method of the neural network described in the above, and will not be described here.
[0151] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of the method. Any reference to memory, storage, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0152] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present disclosure.
[0153] The above-mentioned embodiments only express several embodiments of the present disclosure, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present disclosure, some modifications and improvements can be made, which are within the scope of the present disclosure. Therefore, the protection scope of the present disclosure patent should be subject to the appended claims.
Claims
1. A method of image feature processing of a neural network, characterized by, The method comprises: inputting an original image into a full convolution network of a pre-trained neural network, and outputting a first feature image, wherein the pre-trained neural network comprises: inputting a training image into a neural network, sequentially processing the training image through a full convolution network, a spatial pyramid pooling layer and a Transformer of the neural network, and outputting a processing result of the training image, wherein the training image comprises at least one image; calculating a loss value of the processing result of the training image by using a loss function; obtaining the pre-trained neural network when the loss value is less than or equal to a preset loss threshold value; cutting the first feature image by using an image segmentation method, and generating a first feature image sequence; inputting the first feature image sequence into the spatial pyramid pooling layer of the pre-trained neural network, performing pooling processing on the first feature image sequence by using the spatial pyramid pooling layer, and obtaining a first feature sequence with a fixed dimension; the pooling processing comprises: performing first scale pooling processing on the first feature image sequence by using the spatial pyramid pooling layer, calculating a first average value of a first scale, and taking the first average value as a first scale pooling feature; performing second scale pooling processing on the first feature image sequence by using the spatial pyramid pooling layer, calculating a second average value of a second scale, and taking the second average value as a second scale pooling feature; and calculating the dimension of the feature sequence by using the first scale pooling feature and the second scale pooling feature; inputting the first feature sequence into the Transformer of the pre-trained neural network, and outputting a processing result by using the Transformer.
2. The image feature processing method of neural network according to claim 1, characterized in that, The training image is input into the neural network, sequentially processed through the full convolution network, the spatial pyramid pooling layer and the Transformer of the neural network, and the processing result of the training image is output, which comprises: inputting a second image sequence of the training image into the full convolution network of the neural network, and outputting a second feature image sequence; inputting the second feature image sequence into the spatial pyramid pooling layer of the neural network, performing pooling processing on the second feature image sequence by using the spatial pyramid pooling layer, and obtaining a second feature sequence with a fixed dimension; inputting the second feature sequence with the fixed dimension into the Transformer of the neural network, and outputting the processing result of the training image. 3.The method of claim 1, wherein, Before the second image sequence of the training image is input into the full convolution network of the neural network, the method further comprises: segmenting the training image into a plurality of image blocks by using an image segmentation method, and taking the plurality of image blocks as the second image sequence of the training image.
4. The image feature processing method of neural network according to claim 1, characterized in that, After the loss value of the processing result of the training image is calculated by using the loss function, the method further comprises: updating the weight of the neural network by using a back propagation algorithm when the loss value is greater than the preset loss threshold value; after the weight of the neural network is updated, the pre-training operation is performed again by using the neural network with the updated weight until the loss value is less than or equal to the preset loss threshold value.
5. The image feature processing method of neural network according to claim 3, characterized in that, The second image sequence of the training image is input into the full convolution network of the neural network, which comprises: Input the image blocks in the second image sequence into the first four layers of the convolutional network in the residual network in turn, extract features of the image blocks through the residual network, and output feature image blocks; Combine the feature image blocks to form an image feature sequence.
6. The image feature processing method of neural network according to claim 3, characterized in that, Before the training image is segmented into a plurality of image blocks through the image segmentation method and the plurality of image blocks are taken as a second image sequence of the training image, the method further includes: The training image with the same size is batch trained according to a preset sample number.
7. The image feature processing method of neural networks according to claim 1, characterized in that, After the processing result is output through the Transformer, the method further includes: An image processing task is performed through the processing result, and the image processing task includes image classification, target detection, and semantic segmentation.
8. An image feature processing apparatus of a neural network, characterized by comprising: The device includes: A neural network training module configured to perform a training operation, the training operation including: inputting a training image into a neural network, sequentially processing the training image through a full convolutional network, a spatial pyramid pooling layer, and a Transformer of the neural network, and thereby outputting a processing result of the training image, the training image including at least one image; calculating a loss value of the processing result of the training image through a loss function; and obtaining a pre-trained neural network when the loss value is less than or equal to a preset loss threshold value; A feature image output module configured to input an original image into the full convolutional network of the pre-trained neural network and output a first feature image; An image segmentation module configured to cut the first feature image through an image segmentation method and generate a first feature image sequence; A pooling module configured to input the first feature image sequence into the spatial pyramid pooling layer of the pre-trained neural network, perform pooling processing through the spatial pyramid pooling layer, and obtain a first feature sequence with a fixed dimension; the pooling processing includes: performing first scale pooling processing through the spatial pyramid pooling layer, calculating a first average value of a first scale, and taking the first average value as a first scale pooling feature; performing second scale pooling processing through the spatial pyramid pooling layer, calculating a second average value of a second scale, and taking the second average value as a second scale pooling feature; and calculating a dimension of the feature sequence by using the first scale pooling feature and the second scale pooling feature; A Transformer processing module configured to input the first feature sequence into the Transformer of the pre-trained neural network and output a processing result through the Transformer. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.
11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.