Small target object segmentation method and device based on multi-scale convolution

Abstract

The invention discloses a small target object segmentation method and device based on multi-scale convolution. The method comprises the following steps: preprocessing collected small target images to generate a small target image sample set; training the small target image sample set according to a pre-constructed multi-scale convolutional neural network model and a pre-designed loss function to generate a small target prediction model; and inputting a to-be-predicted small target image into the small target prediction model, and outputting a pixel-level prediction image of the to-be-predicted small target image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a small target object segmentation method and device based on multi-scale convolution. Background Art

[0002] In the field of image processing, the segmentation technology for small target objects has always been an important research direction. Due to the small size of small target objects, the number of pixels in the image is limited and they are easily disturbed by the surrounding background. At the same time, the contrast between small target objects and the surrounding background is low, resulting in unclear boundaries and difficulty in accurately distinguishing between targets and backgrounds. In addition, small target objects are often partially occluded by large target objects or other occluders, making them incomplete in the image, further increasing the difficulty of segmentation. Therefore, solving these challenges is crucial to achieve accurate and robust segmentation of small target objects. Small target object segmentation belongs to the problem of image semantic segmentation. Semantic segmentation is a basic task in computer vision. It involves pixel-by-pixel classification of images and assigning each pixel in the image to the corresponding category. With the development of deep learning, convolutional neural networks (CNNs) can provide more powerful classifiers than traditional classification algorithms and can automatically learn deep features, thereby effectively improving the accuracy of image semantic segmentation. However, traditional CNNs usually use fixed-size convolution kernels and fixed-level network structures in their design, which to some extent limits the network's ability to extract information of different scales. In practical applications, the objects or targets that need to be processed in many scenarios have different scales. For example, in image segmentation tasks, objects of different scales need to be accurately segmented. Traditional CNNs often have difficulty adapting to the processing needs of such multi-scale information. Therefore, in order to process multi-scale information more effectively, researchers have proposed the concept of multi-scale convolution. Multi-scale convolution introduces convolution kernels of different scales or multi-scale convolution operations into the network, so that the network can simultaneously capture and process feature information of different scales. This method effectively expands the network's receptive field and increases the network's adaptability to multi-scale information.

[0003] In the prior art, a Chinese patent with publication number CN115424023B discloses "a self-attention method for enhancing the performance of small target segmentation". This method divides the multi-phase feature map into multiple branches through a self-attention mechanism module for enhancing the performance of small target segmentation, and processes each branch through a single-phase enhanced attention unit to finally obtain the final feature expression. This method can weightedly process channels containing small target feature information, thereby enhancing the ability to segment small targets. However, in complex scenarios, the number of feature maps that need to be processed may be large, resulting in high consumption of computing resources. There are still challenges in distinguishing small targets from the background, resulting in less than ideal segmentation results. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a small target object segmentation method and device based on multi-scale convolution.

[0005] According to one aspect of the present invention, a small target object segmentation method based on multi-scale convolution is provided, comprising:

[0006] Preprocess the collected small target images to generate a small target image sample set;

[0007] The small object image sample set is trained according to the pre-built multi-scale convolutional neural network model and the pre-designed loss function to generate a small object prediction model;

[0008] The small target image to be predicted is input into the small target prediction model, and the pixel-level predicted image of the small target image to be predicted is output; wherein the loss function is:

[0009]

[0010]

[0011]

[0012] In the formula, It is a weighted cross entropy loss function, which is used to measure the difference between the small target object segmentation result output by the multi-scale feature extraction module and the true label; is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module; λ is the weight coefficient used to balance the weighted cross entropy loss function and the Dice loss function; β is the weight parameter for weighting positive samples, where setting β>1 reduces false negatives and setting β<1 reduces false positives; N is the number of categories, is the probability distribution of the model predicting the i-th sample, y i represents the true label of the i-th sample.

[0013] Optionally, the collected small target images are preprocessed to generate a small target image sample set, including:

[0014] Perform denoising operations on small target images;

[0015] Perform image enhancement operations on the denoised small target image;

[0016] Adjust the enhanced small target image according to a preset size;

[0017] The resized small target images are normalized to generate a small target image sample set that meets the preset requirements.

[0018] Optionally, a small object image sample set is trained according to a pre-built multi-scale convolutional neural network model and a pre-designed loss function to generate a small object prediction model, including:

[0019] The small target image set is input into the multi-scale feature extraction module of the multi-scale convolutional neural network model to extract the multi-scale image feature information of each image in the small target image set;

[0020] Input the multi-scale image feature information into the multi-scale feature fusion module of the multi-scale convolutional neural network model, and output the fusion feature;

[0021] Input the fused features into the semantic segmentation prediction module of the multi-scale convolutional neural network model and output the prediction results;

[0022] When the prediction results satisfy the loss function, a small target prediction model is generated.

[0023] Optionally, the multi-scale feature extraction module contains a multi-scale convolution kernel, a depth-separable convolution, a nonlinear activation function, a pooling operation, a feature map pyramid, a skip connection and an attention mechanism, and the small target image set is input into the multi-scale feature extraction module of the multi-scale convolutional neural network model to extract multi-scale image feature information of each image in the small target image set, including:

[0024] By sliding calculation on the images of the small target image set through multi-scale convolution kernel, various feature information in the image is extracted;

[0025] Input multiple feature information into the depth-wise separable convolution to reduce the number of parameters, where the depth-wise separable convolution includes depth-wise convolution and point-wise convolution;

[0026] After the convolution operation, a nonlinear activation function is used to increase the feature representation of multiple feature information;

[0027] Using pooling operations of different sizes will reduce the dimensions of various feature information after adding feature representation;

[0028] Using feature map pyramid, we can capture image features with multiple feature information at different scales.

[0029] Use skip connections to retain image feature information of different scales in image features;

[0030] The attention mechanism is used to strengthen the attention to image feature information of different scales and output multi-scale image feature information.

[0031] Optionally, the multi-scale image feature information is input into a multi-scale feature fusion module of a multi-scale convolutional neural network model, and the fusion features are output, including:

[0032] An upsampling operation is used to rescale the multi-scale image feature information;

[0033] Performing spatial mapping on the scale-adjusted multi-scale image feature information to obtain mapping features;

[0034] Channel attention is introduced, and the mapped features are fused using the splicing method to output the fused features.

[0035] Optionally, the fused features are input into a semantic segmentation prediction module of a multi-scale convolutional neural network model, and prediction results are output, including:

[0036] The number of channels is adjusted to match the number of preset target categories through multiple 1×1 convolutional layers of the semantic segmentation prediction module, and the score of the fused features in each channel is output;

[0037] The Softmax function is used to convert the score of each channel into a probability value, where the sum of the probability values ​​of all channels is 1;

[0038] The Softmax function is used to select the category with the highest probability value as the prediction result.

[0039] Optionally, the expression of the Softmax function is as follows:

[0040]

[0041] Among them, z i represents the i-th element in the input vector, e is the base of the natural logarithm, and K is the number of categories.

[0042] According to another aspect of the present invention, a small target object segmentation device based on multi-scale convolution is provided, comprising:

[0043] The first generation module is used to pre-process the collected small target images to generate a small target image sample set;

[0044] The second generation module is used to train the small object image sample set according to the pre-built multi-scale convolutional neural network model and the pre-designed loss function to generate a small object prediction model;

[0045] The output module is used to input the small target image to be predicted into the small target prediction model and output the pixel-level predicted image of the small target image to be predicted; wherein the loss function is:

[0046]

[0047]

[0048] In the formula, It is a weighted cross entropy loss function, which is used to measure the difference between the small target object segmentation result output by the multi-scale feature extraction module and the true label; is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module; λ is the weight coefficient used to balance the weighted cross entropy loss function and the Dice loss function; β is the weight parameter for weighting positive samples, where setting β>1 reduces false negatives and setting β<1 reduces false positives; N is the number of categories, is the probability distribution of the model predicting the i-th sample, y i represents the true label of the i-th sample.

[0049] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0050] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0051] Thus, the present invention utilizes a multi-scale convolutional network to extract features at different scales, which can more comprehensively capture the various forms and details of small target objects, thereby improving the model's ability to segment small target objects of different scales. This helps to improve the performance of the model in complex scenes. The introduction of the attention mechanism allows the model to pay more attention to the important areas around the small target objects, thereby improving the recognition accuracy of the model. This helps to reduce errors and improve the reliability of the model in actual scenes. Thus, the present invention will help to improve the accuracy and robustness of the segmentation of small target objects, thereby improving the overall performance. The comprehensive use of multi-scale feature extraction and attention mechanism can better cope with the challenges of small target segmentation in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0053] Figure 1 is a flowchart of a small target object segmentation method based on multi-scale convolution provided by an exemplary embodiment of the present invention;

[0054] Figure 2 is another flowchart of a small target object segmentation method based on multi-scale convolution provided by an exemplary embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of a multi-scale convolutional neural network of a small target object segmentation method based on multi-scale convolution provided by an exemplary embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of a multi-scale feature extraction module of a small target object segmentation method based on multi-scale convolution provided by an exemplary embodiment of the present invention;

[0057] Figure 5 is a schematic diagram of a multi-scale feature fusion module of a small target object segmentation method based on multi-scale convolution provided by an exemplary embodiment of the present invention;

[0058] Figure 6 is a structural schematic diagram of a small target object segmentation device based on multi-scale convolution provided by an exemplary embodiment of the present invention;

[0059] Figure 7 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0060] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0061] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0062] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0063] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0064] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0065] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0066] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0067] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0068] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0069] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0070] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0071] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0072] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0073] Exemplary Methods

[0074] Figure 1 FIG. 1 is a flow chart of a small target object segmentation method based on multi-scale convolution provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the small target object segmentation method 100 based on multi-scale convolution includes the following steps:

[0075] Step 101, preprocessing the collected small target images to generate a small target image sample set;

[0076] Step 102, training the small object image sample set according to the pre-built multi-scale convolutional neural network model and the pre-designed loss function to generate a small object prediction model;

[0077] Step 103, input the small target image to be predicted into the small target prediction model, and output the pixel-level predicted image of the small target image to be predicted; wherein the loss function is:

[0078]

[0079]

[0080]

[0081] In the formula, It is a weighted cross entropy loss function, which is used to measure the difference between the small target object segmentation result output by the multi-scale feature extraction module and the true label; is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module; λ is the weight coefficient used to balance the weighted cross entropy loss function and the Dice loss function; β is the weight parameter for weighting positive samples, where setting β>1 reduces false negatives and setting β<1 reduces false positives; N is the number of categories, is the probability distribution of the model predicting the i-th sample, y irepresents the true label of the i-th sample.

[0082] Specifically, the present invention discloses a small target object segmentation method based on multi-scale convolution, which accurately segments small target objects through a multi-scale convolutional neural network semantic segmentation model. The small target object segmentation method comprises the following steps: first, a multi-scale convolutional neural network semantic segmentation model suitable for parsing small target images is designed, then a loss function is designed, a strategy for adjusting parameters during training is determined, and a small target image parsing model is subsequently obtained, and the model can parse the input image into a pixel-level predicted image. Finally, necessary post-processing is performed on the model output, such as removing small area noise, edge smoothing and other operations to improve the quality of the segmentation result. The segmentation model comprises a preprocessing module, a multi-scale feature extraction module, a multi-scale feature fusion module and a semantic segmentation prediction module. The multi-scale feature extraction module comprises a plurality of convolutional layers, and convolution kernels of different scales are used for feature extraction. The multi-scale feature fusion module fuses feature maps of different resolutions to obtain a richer and more accurate feature representation, and finally outputs an accurate small target object segmentation result. The present invention can significantly improve the accuracy and speed of small target segmentation by improving these algorithms.

[0083] The present invention provides a small target object segmentation method based on multi-scale convolution, comprising the following steps:

[0084] Step 1: Design a multi-scale convolutional neural network semantic segmentation model suitable for parsing small object images;

[0085] Step 2: Train the designed multi-scale convolutional neural network model;

[0086] Step 3: Use the trained model to parse the image and generate a pixel-level predicted image.

[0087] Furthermore, the model in step 1 comprises four modules, namely, a preprocessing module, a multi-scale feature extraction module, a multi-scale feature fusion module and a semantic segmentation prediction module. It includes the following sub-steps:

[0088] (1.1) Preprocess the collected small target image data.

[0089] (1.2) The preprocessed image is sent to the multi-scale feature extraction module to extract image features.

[0090] (1.3) The feature information of different scales extracted by the multi-scale feature extraction module is sent to the multi-scale feature fusion module.

[0091] (1.4) The result of feature fusion is sent to the semantic segmentation prediction module to obtain the prediction result.

[0092] Furthermore, the preprocessing module contains image denoising, image enhancement, resizing and normalization. It includes the following sub-steps:

[0093] (1.1.1) Perform denoising on the acquired image to improve the accuracy of subsequent processing.

[0094] (1.1.2) Perform image enhancement operations, such as adjusting contrast, brightness, color balance, etc., to improve image quality and model robustness.

[0095] (1.1.3) Resize the image to the size required for model input.

[0096] (1.1.4) Normalize the image and scale the pixel values ​​to the range [0,1].

[0097] Furthermore, the multi-scale feature extraction module contains multi-scale convolution kernels, depth-wise separable convolution, nonlinear activation function, pooling operation, feature map pyramid, skip connection and attention mechanism. It includes the following sub-steps:

[0098] (1.2.1) Various feature information in the image is extracted by sliding the convolution kernel on the image. These convolution kernels have receptive fields of different sizes and are used to capture feature information of different scales in the image.

[0099] (1.2.2) The standard convolution operation is decomposed into two steps: depthwise convolution and pointwise convolution. Depthwise separable convolution can reduce the number of parameters and reduce computational cost.

[0100] (1.2.3) After the convolution operation, a nonlinear activation function is applied to increase the nonlinear expression ability of the network and introduce richer feature representation.

[0101] (1.2.4) Use pooling kernels of different sizes to extract multi-scale features. The pooling operation is used to reduce the dimension of the feature map, reduce the amount of calculation, and retain the most important feature information.

[0102] (1.2.5) Construct a multi-scale feature map pyramid to capture rich image features at different scales.

[0103] (1.2.6) Use skip connections to help retain feature information of different scales. The introduction of skip connections can establish direct connections between low-level features and high-level features, which helps to reduce information loss and retain more detailed information, and improve the accuracy and robustness of the model.

[0104] (1.2.7) The attention mechanism is introduced to strengthen the focus on features of different scales. The model pays more attention to important features to reduce information loss. By giving different weights to different feature maps, the perception of key features can be improved.

[0105] Furthermore, the multi-scale feature fusion module contains scale adaptation, feature mapping, feature fusion and channel attention. It includes the following sub-steps:

[0106] (1.3.1) Before feature fusion, feature maps of different scales are scale-adapted and upsampling operations are used to adjust the size of feature maps.

[0107] (1.3.2) Map the adjusted features so as to perform fusion operations in the feature space.

[0108] (1.3.3) Use the splicing method to fuse the mapped features.

[0109] (1.3.4) In the feature fusion process, channel attention is introduced to dynamically adjust the importance of different channels.

[0110] Furthermore, the semantic segmentation prediction module includes a 1×1 convolutional layer, a Softmax activation function, and a prediction output. It includes the following sub-steps:

[0111] (1.4.1) The semantic segmentation prediction module starts with multiple 1×1 convolutional layers, which adjust the number of channels of the feature map to match the number of target categories.

[0112] (1.4.2) The Softmax function converts each score into a probability value between 0 and 1 and ensures that the sum of the probabilities of all categories is 1. The expression of the Softmax function is as follows:

[0113]

[0114] Among them, z i represents the i-th element in the input vector, e is the base of the natural logarithm, and K is the number of categories. The calculation process of the Softmax function is to take the exponential of each element of the input vector Then It means to sum all the index values, and finally divide the index value of each element by this sum to get the probability of each category.

[0115] (1.4.3) The final output is to generate the predicted segmentation result by selecting the category with the highest probability through the Softmax function.

[0116] Furthermore, the loss function formula used in the training process of step 2 is as follows;

[0117]

[0118]

[0119]

[0120] in It is a weighted cross entropy loss function, which is used to measure the difference between the small target object segmentation result output by the multi-scale feature extraction module and the true label. is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module. It measures the overlap between the predicted results and the true results. λ is the weight coefficient used to balance the weighted cross entropy loss and the Dice loss, which is adjusted according to the sample distribution. β is the weight parameter for weighting positive samples. Setting β>1 reduces false negatives; setting β<1 reduces false positives. N is the number of categories, is the probability distribution of the model predicting the i-th sample, y i represents the true label of the i-th sample.

[0121] In one embodiment of the present invention, the present invention provides a small target object segmentation method based on multi-scale convolution. Figure 2 , Figure 2 This is a technical solution flow chart of a small target object segmentation method based on multi-scale convolution provided by the implementation use case, which specifically includes the following steps:

[0122] (1) Design a multi-scale convolutional neural network semantic segmentation model suitable for parsing small object images;

[0123] The multi-scale convolutional neural network model includes a preprocessing module, a multi-scale feature extraction module, a multi-scale feature fusion module and a semantic segmentation prediction module. Figure 3 , Figure 3 Schematic diagram of a multi-scale convolutional neural network of a small target object segmentation method based on multi-scale convolution provided in an embodiment. Specifically, the method comprises the following steps:

[0124] (1.1) Preprocess the collected small target image data.

[0125] (1.2) The preprocessed image is sent to the multi-scale feature extraction module to extract image features.

[0126] (1.3) The feature information of different scales extracted by the multi-scale feature extraction module is sent to the multi-scale feature fusion module.

[0127] (1.4) The result of feature fusion is sent to the semantic segmentation prediction module to obtain the prediction result.

[0128] The preprocessing module includes image denoising, image enhancement, resizing and normalization. It includes the following sub-steps:

[0129] (1.1.1) Perform denoising on the acquired image to improve the accuracy of subsequent processing.

[0130] (1.1.2) Perform image enhancement operations, such as adjusting contrast, brightness, color balance, etc., to improve image quality and model robustness.

[0131] (1.1.3) Resize the image to the size required for model input.

[0132] (1.1.4) Normalize the image and scale the pixel values ​​to the range [0,1].

[0133] The multi-scale feature extraction module contains multi-scale convolution kernels, depth-wise separable convolution, nonlinear activation function, pooling operation, feature map pyramid, skip connection and attention mechanism. Figure 4 , Figure 4 Schematic diagram of a multi-scale feature extraction module of a small target object segmentation method based on multi-scale convolution provided in an embodiment. The method includes the following sub-steps:

[0134] (1.2.1) Various feature information in the image is extracted by sliding the convolution kernel on the image. These convolution kernels have receptive fields of different sizes and are used to capture feature information of different scales in the image.

[0135] (1.2.2) The standard convolution operation is decomposed into two steps: depthwise convolution and pointwise convolution. Depthwise separable convolution can reduce the number of parameters and reduce computational cost.

[0136] (1.2.3) After the convolution operation, a nonlinear activation function is applied to increase the nonlinear expression ability of the network and introduce richer feature representation.

[0137] (1.2.4) Use pooling kernels of different sizes to extract multi-scale features. The pooling operation is used to reduce the dimension of the feature map, reduce the amount of calculation, and retain the most important feature information.

[0138] (1.2.5) Construct a multi-scale feature map pyramid to capture rich image features at different scales.

[0139] (1.2.6) Use skip connections to help retain feature information of different scales. The introduction of skip connections can establish direct connections between low-level features and high-level features, which helps to reduce information loss and retain more detailed information, and improve the accuracy and robustness of the model.

[0140] (1.2.7) The attention mechanism is introduced to strengthen the focus on features of different scales. The model pays more attention to important features to reduce information loss. By giving different weights to different feature maps, the perception of key features can be improved.

[0141] The multi-scale feature fusion module includes scale adaptation, feature mapping, feature fusion and channel attention. Figure 5 , Figure 5 Schematic diagram of a multi-scale feature fusion module of a small target object segmentation method based on multi-scale convolution provided in an embodiment. The method includes the following sub-steps:

[0142] (1.3.1) Before feature fusion, feature maps of different scales are scale-adapted and upsampling operations are used to adjust the size of feature maps.

[0143] (1.3.2) Map the adjusted features so as to perform fusion operations in the feature space.

[0144] (1.3.3) Use the splicing method to fuse the mapped features.

[0145] (1.3.4) In the feature fusion process, channel attention is introduced to dynamically adjust the importance of different channels.

[0146] Among them, the semantic segmentation prediction module consists of a 1×1 convolutional layer, a Softmax activation function and prediction output.

[0147] (1.4.1) The prediction module starts with multiple 1×1 convolutional layers, which adjust the number of channels of the feature map to match the number of target categories.

[0148] (1.4.2) The Softmax function converts each score into a probability value between 0 and 1 and ensures that the sum of the probabilities of all categories is 1. The expression of the Softmax function is as follows:

[0149]

[0150] Among them, z i represents the i-th element in the input vector, e is the base of the natural logarithm, and K is the number of categories. The calculation process of the Softmax function is to take the exponential of each element of the input vector Then It means to sum all the index values, and finally divide the index value of each element by this sum to get the probability of each category.

[0151] (1.4.3) The final output is to generate the predicted segmentation result by selecting the category with the highest probability through the Softmax function.

[0152] (2) Train the designed multi-scale convolutional neural network model;

[0153] Specifically, the loss function formula used in the training process is as follows:

[0154]

[0155]

[0156]

[0157] in It is a weighted cross entropy loss function, which is used to measure the difference between the small target object segmentation result output by the multi-scale feature extraction module and the true label. is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module. It measures the overlap between the predicted results and the true results. λ is the weight coefficient used to balance the weighted cross entropy loss and the Dice loss, which is adjusted according to the sample distribution. β is the weight parameter for weighting positive samples. Setting β>1 reduces false negatives; setting β<1 reduces false positives. N is the number of categories, is the probability distribution of the model predicting the i-th sample, y i represents the true label of the i-th sample.

[0158] (3) Use the trained model to parse the image and generate pixel-level predicted images.

[0159] Specifically, by loading the pre-trained multi-scale convolutional neural network segmentation model, image preprocessing and model inference are performed on the small target image to be analyzed, and pixel-level semantic segmentation prediction is generated. The model output is then post-processed as necessary, such as removing small area noise and edge smoothing, to improve the quality of the segmentation result. Finally, the segmentation result can be visualized or saved to obtain a fine semantic segmentation of the image.

[0160] Thus, the present invention utilizes a multi-scale convolutional network to extract features at different scales, which can more comprehensively capture the various forms and details of small target objects, thereby improving the model's ability to segment small target objects of different scales. This helps to improve the performance of the model in complex scenes. The introduction of the attention mechanism allows the model to pay more attention to the important areas around the small target objects, thereby improving the recognition accuracy of the model. This helps to reduce errors and improve the reliability of the model in actual scenes. Thus, the present invention will help to improve the accuracy and robustness of the segmentation of small target objects, thereby improving the overall performance. The comprehensive use of multi-scale feature extraction and attention mechanism can better cope with the challenges of small target segmentation in complex scenes.

[0161] Exemplary Devices

[0162] Figure 6 FIG. 1 is a schematic diagram of a small target object segmentation device based on multi-scale convolution provided by an exemplary embodiment of the present invention. Figure 6 As shown, the device 600 includes:

[0163] A first generating module 610 is used to pre-process the collected small object images to generate a small object image sample set;

[0164] A second generation module 620 is used to train a small object image sample set according to a pre-built multi-scale convolutional neural network model and a pre-designed loss function to generate a small object prediction model;

[0165] The output module 630 is used to input the small target image to be predicted into the small target prediction model and output the pixel-level predicted image of the small target image to be predicted; wherein the loss function is:

[0166]

[0167]

[0168]

[0169] In the formula, It is a weighted cross entropy loss function, which is used to measure the difference between the small target object segmentation result output by the multi-scale feature extraction module and the true label; is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module; λ is the weight coefficient used to balance the weighted cross entropy loss function and the Dice loss function; β is the weight parameter for weighting positive samples, where setting β>1 reduces false negatives and setting β<1 reduces false positives; N is the number of categories, is the probability distribution of the model predicting the i-th sample, y i represents the true label of the i-th sample.

[0170] Optionally, the first generating module 610 includes:

[0171] Denoising submodule, used to perform denoising operations on small target images;

[0172] The enhancement submodule is used to perform image enhancement operations on the denoised small target image;

[0173] An adjustment submodule, used for adjusting the enhanced small target image according to a preset size;

[0174] The first generating submodule is used to normalize the resized small target image to generate a small target image sample set that meets preset requirements.

[0175] Optionally, the second generating module 620 includes:

[0176] An extraction submodule, used for inputting the small object image set into the multi-scale feature extraction module of the multi-scale convolutional neural network model to extract the multi-scale image feature information of each image in the small object image set;

[0177] A first output submodule is used to input the multi-scale image feature information into the multi-scale feature fusion module of the multi-scale convolutional neural network model, and output the fusion feature;

[0178] The second output submodule is used to input the fused features into the semantic segmentation prediction module of the multi-scale convolutional neural network model and output the prediction results;

[0179] The second generation submodule is used to generate a small target prediction model when the prediction result satisfies the loss function.

[0180] Optionally, the multi-scale feature extraction module contains a multi-scale convolution kernel, a depth-separable convolution, a non-linear activation function, a pooling operation, a feature map pyramid, a skip connection and an attention mechanism, and an extraction submodule, including:

[0181] A computing unit, used to perform sliding calculations on the images of the small target image set through a multi-scale convolution kernel to extract various feature information in the image;

[0182] A reduction unit, used for inputting a variety of feature information into a depthwise separable convolution to reduce the amount of parameters, wherein the depthwise separable convolution includes a depthwise convolution and a pointwise convolution;

[0183] An adding unit is used to add feature representation of multiple feature information using nonlinear activation functions after the convolution operation;

[0184] A representation unit is used to reduce the dimensions of various feature information after adding feature representation using pooling operations of different sizes;

[0185] A capture unit, used to capture image features of multiple feature information at different scales using a feature map pyramid;

[0186] A retention unit, used to retain image feature information of different scales in the image features by using skip connections;

[0187] The first output unit is used to use the attention mechanism to strengthen the attention to image feature information of different scales and output multi-scale image feature information.

[0188] Optionally, the first output submodule includes:

[0189] An adjustment unit is used to adjust the scale of the multi-scale image feature information by upsampling operation; an acquisition unit is used to perform spatial mapping on the multi-scale image feature information after the scale adjustment to obtain mapping features;

[0190] The fusion unit is used to introduce channel attention, fuse the mapped features by splicing method, and output fused features.

[0191] Optionally, the second output submodule includes:

[0192] A second output unit is used to adjust the number of channels to a number of channels that matches the number of preset target categories through multiple 1×1 convolutional layers of the semantic segmentation prediction module, and output the score of the fused features in each channel;

[0193] A conversion unit, used to convert the score of each channel into a probability value using a Softmax function, where the sum of the probability values ​​of all channels is 1;

[0194] As a unit, it is used to select the category with the highest probability value as the prediction result through the Softmax function.

[0195] Optionally, the expression of the Softmax function is as follows:

[0196]

[0197] Among them, z i represents the i-th element in the input vector, e is the base of the natural logarithm, and K is the number of categories.

[0198] Exemplary Electronic Devices

[0199] Figure 7 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 7 As shown, the electronic device 70 includes one or more processors 71 and a memory 72 .

[0200] The processor 71 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0201] The memory 72 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 71 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 73 and an output device 74, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0202] In addition, the input device 73 may also include, for example, a keyboard, a mouse, and the like.

[0203] The output device 74 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0204] Of course, to simplify, Figure 7 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device may also include any other appropriate components.

[0205] Exemplary computer program products and computer-readable storage media

[0206] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0207] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0208] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0209] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0210] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0211] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0212] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0213] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0214] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0215] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A small target object segmentation method based on multi-scale convolution, It is characterized in that include: Preprocess the collected small target images to generate a small target image sample set; The small object image sample set is trained according to a pre-built multi-scale convolutional neural network model and a pre-designed loss function to generate a small object prediction model; The small target image to be predicted is input into the small target prediction model, and a pixel-level predicted image of the small target image to be predicted is output; wherein the loss function is: In the formula, It is a weighted cross entropy loss function, which is used to measure the difference between the segmentation result of small target objects output by the multi-scale feature extraction module and the true label; is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module; λ is the weight coefficient used to balance the weighted cross entropy loss function and the Dice loss function; β is the weight parameter for weighting positive samples, where setting β>1 reduces false negatives and setting β<1 reduces false positives; N is the number of categories, is the probability distribution of the model predicting the i-th sample, y i represents the true label of the i-th sample.

2. The method according to claim 1, It is characterized in that The collected small target images are preprocessed to generate a small target image sample set, including: Performing a denoising operation on the small target image; Perform image enhancement operations on the denoised small target image; Adjust the enhanced small target image according to a preset size; The resized small target images are normalized to generate the small target image sample set that meets preset requirements.

3. The method according to claim 1, It is characterized in that The small object image sample set is trained according to a pre-built multi-scale convolutional neural network model and a pre-designed loss function to generate a small object prediction model, including: Inputting the small object image set into the multi-scale feature extraction module of the multi-scale convolutional neural network model to extract multi-scale image feature information of each image in the small object image set; Inputting the multi-scale image feature information into a multi-scale feature fusion module of the multi-scale convolutional neural network model, and outputting fusion features; Inputting the fused features into the semantic segmentation prediction module of the multi-scale convolutional neural network model, and outputting the prediction results; When the prediction result satisfies the loss function, the small target prediction model is generated.

4. The method according to claim 3, It is characterized in that The multi-scale feature extraction module contains a multi-scale convolution kernel, a depth-separable convolution, a nonlinear activation function, a pooling operation, a feature map pyramid, a jump connection and an attention mechanism. The small target image set is input into the multi-scale feature extraction module of the multi-scale convolutional neural network model to extract the multi-scale image feature information of each image in the small target image set, including: Performing sliding calculation on the images of the small target image set by the multi-scale convolution kernel to extract various feature information in the images; Inputting a variety of feature information into the depthwise separable convolution to reduce the amount of parameters, wherein the depthwise separable convolution includes depthwise convolution and pointwise convolution; After the convolution operation, the nonlinear activation function is used to increase the feature representation of multiple feature information; Using pooling operations of different sizes will reduce the dimensions of various feature information after adding feature representation; Using the feature map pyramid, image features of various feature information at different scales are captured; Using the skip connection to retain image feature information of different scales in the image features; The attention mechanism is used to strengthen the attention to image feature information of different scales, and the multi-scale image feature information is output.

5. The method according to claim 3, It is characterized in that Inputting the multi-scale image feature information into the multi-scale feature fusion module of the multi-scale convolutional neural network model, and outputting fusion features, including: Using an upsampling operation to rescale the multi-scale image feature information; Performing spatial mapping on the scale-adjusted multi-scale image feature information to obtain mapping features; Channel attention is introduced, and the mapped features are fused by a splicing method to output the fused features.

6. The method according to claim 3, It is characterized in that Inputting the fusion features into the semantic segmentation prediction module of the multi-scale convolutional neural network model and outputting the prediction results includes: Adjusting the number of channels to a number of channels that matches the number of preset target categories through multiple 1×1 convolutional layers of the semantic segmentation prediction module, and outputting the score of the fused feature in each channel; The Softmax function is used to convert the score of each channel into a probability value, where the sum of the probability values ​​of all channels is 1; The category with the highest probability value is selected as the prediction result through the Softmax function.

7. The method according to claim 6, It is characterized in that The expression of the Softmax function is as follows: Among them, z i represents the i-th element in the input vector, e is the base of the natural logarithm, and K is the number of categories.

8. A small target object segmentation device based on multi-scale convolution, It is characterized in that include: The first generation module is used to preprocess the collected small target images to generate a small target image sample set; A second generation module is used to train the small object image sample set according to a pre-built multi-scale convolutional neural network model and a pre-designed loss function to generate a small object prediction model; The output module is used to input the small target image to be predicted into the small target prediction model and output the pixel-level predicted image of the small target image to be predicted; wherein the loss function is: In the formula, It is a weighted cross entropy loss function, which is used to measure the difference between the small target object segmentation result output by the multi-scale feature extraction module and the true label; is the Dice loss function, which is used to enhance the accuracy of the small target object segmentation results output by the multi-scale feature extraction module; λ is the weight coefficient used to balance the weighted cross entropy loss function and the Dice loss function; β is the weight parameter for weighting positive samples, where setting β>1 reduces false negatives and setting β<1 reduces false positives; N is the number of categories, is the probability distribution of the model predicting the i-th sample, y i represents the true label of the i-th sample.

9. The device according to claim 8, It is characterized in that The first generation module includes: A denoising submodule, used for performing a denoising operation on the small target image; The enhancement submodule is used to perform image enhancement operations on the denoised small target image; An adjustment submodule, used for adjusting the enhanced small target image according to a preset size; The first generating submodule is used to normalize the resized small target image to generate the small target image sample set that meets the preset requirements.

10. The device according to claim 8, It is characterized in that The second generation module includes: An extraction submodule, used for inputting the small object image set into the multi-scale feature extraction module of the multi-scale convolutional neural network model to extract multi-scale image feature information of each image in the small object image set; A first output submodule, used for inputting the multi-scale image feature information into the multi-scale feature fusion module of the multi-scale convolutional neural network model, and outputting fusion features; A second output submodule, used for inputting the fusion features into the semantic segmentation prediction module of the multi-scale convolutional neural network model, and outputting the prediction results; The second generating submodule is used to generate the small target prediction model when the prediction result satisfies the loss function.

11. The device according to claim 10, It is characterized in that The multi-scale feature extraction module contains multi-scale convolution kernels, depth-separable convolution, nonlinear activation function, pooling operation, feature map pyramid, skip connection and attention mechanism, and extracts sub-modules, including: A computing unit, configured to perform sliding calculations on the images of the small target image set by using the multi-scale convolution kernel to extract a variety of feature information in the images; A reduction unit, configured to input a plurality of feature information into the depthwise separable convolution to reduce the amount of parameters, wherein the depthwise separable convolution includes depthwise convolution and pointwise convolution; An adding unit, used for adding feature representations of multiple feature information using the nonlinear activation function after the convolution operation; A representation unit is used to reduce the dimensions of various feature information after adding feature representation using pooling operations of different sizes; A capture unit, used to capture image features of multiple feature information at different scales using the feature map pyramid; A retaining unit, used for retaining image feature information of different scales in the image features by using the skip connection; The first output unit is used to use the attention mechanism to strengthen the attention to image feature information of different scales and output the multi-scale image feature information.

12. The device according to claim 10, It is characterized in that The first output submodule includes: An adjustment unit, configured to adjust the scale of the multi-scale image feature information by using an upsampling operation; An acquisition unit, configured to perform spatial mapping on the multi-scale image feature information after scale adjustment to acquire mapping features; The fusion unit is used to introduce channel attention, fuse the mapped features by a splicing method, and output the fused features.

13. The device according to claim 10, It is characterized in that The second output submodule includes: A second output unit, used for adjusting the number of channels to a number of channels matching a preset number of target categories through a plurality of 1×1 convolutional layers of the semantic segmentation prediction module, and outputting a score of the fused feature in each channel; A conversion unit, used to convert the score of each channel into a probability value using a Softmax function, where the sum of the probability values ​​of all channels is 1; As a unit, it is used to select the category with the highest probability value as the prediction result through the Softmax function.

14. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

15. An electronic device, It is characterized in that The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 7.

Images