Target extraction method based on remote sensing image and training method of target extraction model

Abstract

The present disclosure provides a target extraction method based on remote sensing images and a training method for a target extraction model, which can be applied to the fields of image processing and target extraction. The target extraction method includes: performing N rounds of processing on M first feature matrices corresponding to an image to be identified to obtain N coding feature matrices, wherein the image to be identified includes multiple target identification objects, the resolution sizes of the M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer; performing feature enhancement processing on the N coding feature matrices to obtain N enhanced feature matrices; performing decoding processing on the N enhanced feature matrices to obtain a binary image with the same resolution size as the image to be identified; and determining a target extraction result based on the number of pixels of the target identification object in the binary image and the spatial resolution of the image to be identified, wherein the target extraction result represents the number of target identification objects.

Description

Technical Field

[0001] The present disclosure relates to the fields of image processing and target extraction, and in particular to a target extraction method based on remote sensing images and a training method for a target extraction model. Background Art

[0002] With the continuous development of high spatial resolution remote sensing technology and the rapid iteration of deep learning technology, more and more remote sensing images collected by satellite systems, such as Beijing-3, Sustainable Development Goals Satellite-1 (SDGSAT-1) and resource satellites, have been widely used in target monitoring tasks in various fields and have achieved fruitful results. In the task of monitoring sheep based on remote sensing images, since sheep are mostly white in remote sensing images, which is obviously different from the background color, the color threshold segmentation method is usually used to extract the distribution of sheep in remote sensing images.

[0003] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: in scenes with more complex backgrounds, factors such as insufficient lighting can easily lead to a low contrast between the color of the flock and the background color, so that the color threshold segmentation method is difficult to accurately obtain the distribution of the flock. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a target extraction method based on remote sensing images and a training method for a target extraction model.

[0005] According to a first aspect of the present disclosure, there is provided a target extraction method, comprising:

[0006] Performing N rounds of processing on M first feature matrices corresponding to the image to be identified to obtain N encoding feature matrices, wherein the image to be identified includes a plurality of target identification objects, the resolutions of the M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer;

[0007] Performing feature enhancement processing on the above N encoding feature matrices to obtain N enhanced feature matrices;

[0008] Decoding the N enhanced feature matrices to obtain a binary image having the same resolution as the image to be identified; and

[0009] Based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized, a target extraction result is determined, wherein the target extraction result represents the number of the target recognition object.

[0010] According to an embodiment of the present disclosure, the above-mentioned N rounds of processing on the M first feature matrices corresponding to the image to be recognized to obtain N encoded feature matrices include:

[0011] In the case of n = 1, perform feature fusion processing on each of the M first feature matrices in the first round to obtain M fused feature matrices in the first round;

[0012] Perform context enhancement processing on each of the M first feature matrices in the first round to obtain M context-enhanced matrices in the first round;

[0013] Fuse the above-mentioned M fused feature matrices and the above-mentioned M context-enhanced matrices to obtain M second feature matrices in the second round;

[0014] In the case of 1 < n ≤ N,

[0015] Perform feature fusion processing on each of the M second feature matrices in the (n - 1)th round to obtain M fused feature matrices in the (n - 1)th round;

[0016] Perform context enhancement processing on each of the M second feature matrices in the (n - 1)th round to obtain M context-enhanced matrices in the (n - 1)th round;

[0017] Fuse the above-mentioned M fused feature matrices and the context-enhanced matrices corresponding to each of the above-mentioned M fused feature matrices respectively to obtain M second feature matrices in the nth round;

[0018] Perform resampling operations on the above-mentioned M second feature matrices in the nth round to obtain M resampled feature matrices; and

[0019] Fuse the above-mentioned M resampled feature matrices to obtain the encoded feature matrix in the (n - 1)th round.

[0020] According to an embodiment of the present disclosure, the above-mentioned performing feature fusion processing on each of the M second feature matrices in the (n - 1)th round to obtain M fused feature matrices in the (n - 1)th round includes:

[0021] Perform convolution processing on the i-th second feature matrix in the (n - 1)th round to obtain the i-th convolution feature matrix, where i is a positive integer less than or equal to M;

[0022] Based on the above-mentioned (M - 1) second feature matrices in the (n - 1)th round, determine (M - 1) to-be-fused feature matrices, where the above-mentioned (M - 1) to-be-fused feature matrices have the same resolution size as the above-mentioned i-th convolution feature matrix; and

[0023] Fuse the above-mentioned i-th convolution feature matrix with the above-mentioned (M - 1) to-be-fused feature matrices to obtain the i-th fused feature matrix.

[0024] According to an embodiment of the present disclosure, context enhancement processing is respectively performed on the M second feature matrices of the (n - 1)th round to obtain M context enhancement matrices of the (n - 1)th round, including:

[0025] Feature fusion is performed on the M second feature matrices of the (n - 1)th round to obtain a context enhancement matrix to be enhanced of the (n - 1)th round, wherein the number of channels of the context enhancement matrix to be enhanced is the sum of the numbers of channels of the M second feature matrices of the (n - 1)th round respectively;

[0026] The context enhancement matrix to be enhanced of the (n - 1)th round is input into the channel attention module of the (n - 1)th round to obtain a channel enhanced feature matrix of the (n - 1)th round, wherein the channel attention module of the (n - 1)th round is used to adjust the weights of multiple channels of the context enhancement matrix to be enhanced of the (n - 1)th round; and

[0027] Resampling is performed on the channel enhanced feature matrix of the (n - 1)th round to obtain M context enhancement matrices, wherein the matrix sizes of the M context enhancement matrices of the (n - 1)th round correspond one-to-one to the resolution sizes of the M fusion feature matrices of the (n - 1)th round.

[0028] According to an embodiment of the present disclosure, feature enhancement processing is performed on the N encoded feature matrices to obtain N enhanced feature matrices, including:

[0029] Downsampling operations are performed on the first (j - 1) encoded feature matrices to obtain (j - 1) downsampled feature matrices;

[0030] The (j - 1) downsampled feature matrices are fused with the jth encoded feature matrix to obtain a jth initial enhanced feature matrix, wherein the resolution of the jth encoded feature matrix is less than the resolution of the first (j - 1) encoded feature matrices, 1 < j ≤ N and j is an integer; and

[0031] The jth initial enhanced feature matrix is input into the enhanced channel attention module corresponding to the jth encoded feature matrix to obtain a jth enhanced feature matrix, wherein the enhanced channel attention module corresponding to the jth encoded feature matrix is used to adjust the weights of multiple channels of the jth initial enhanced feature matrix.

[0032] According to an embodiment of the present disclosure, decoding processing is performed on the N enhanced feature matrices to obtain a binary image having the same resolution size as the image to be recognized, including:

[0033] The N enhanced feature matrices are fused using a multi-scale feature fusion technique to obtain a multi-scale fusion matrix;

[0034] Resample the above multi-scale fusion matrix to obtain a reconstructed feature matrix;

[0035] The reconstructed feature matrix is ​​processed by using an activation function to obtain a threshold feature matrix, wherein each pixel value in the threshold feature matrix is ​​between 0 and 1;

[0036] The threshold feature matrix is ​​binarized according to a preset pixel threshold to obtain the binary image.

[0037] According to an embodiment of the present disclosure, the target extraction result is determined based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized, including:

[0038] The binary image is processed by using an image segmentation algorithm to obtain the number of pixels of the target recognition object in the binary image;

[0039] Determine the area occupied by the target object based on the spatial resolution of the image to be identified and the number of pixels;

[0040] Based on the above-mentioned occupied area and the average density of the target identification objects, the number of the above-mentioned target identification objects is determined.

[0041] A second aspect of the present disclosure provides a method for training a target extraction model, the method comprising:

[0042] Performing N rounds of processing on M first sample feature matrices corresponding to the sample image to be identified to obtain N coded sample feature matrices, wherein the sample image to be identified includes a plurality of target identification objects, the sample image to be identified corresponds to label information, the resolutions of the M first sample feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer;

[0043] Performing feature enhancement processing on the N encoded sample feature matrices to obtain N enhanced sample feature matrices;

[0044] Decoding the N enhanced sample feature matrices to obtain a sample binary image having the same resolution as the image to be identified; and

[0045] Using the above label information and the above sample binary images, a deep learning model is trained to obtain a target extraction model.

[0046] A third aspect of the present disclosure provides a target extraction device, the device comprising:

[0047] A first encoding module is used to perform N rounds of processing on M first feature matrices corresponding to the image to be identified to obtain N encoding feature matrices, wherein the image to be identified includes a plurality of target identification objects, the resolutions of the M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer;

[0048] A first enhancement module is used to perform feature enhancement processing on the N encoding feature matrices to obtain N enhanced feature matrices;

[0049] A first decoding module is used to decode the N enhanced feature matrices to obtain a binary image with the same resolution as the image to be identified; and

[0050] The result determination module is used to determine the target extraction result based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized, wherein the target extraction result represents the number of the target recognition object.

[0051] A fourth aspect of the present disclosure provides a training device for a target extraction model, the device comprising:

[0052] A second encoding module is used to perform N rounds of processing on M first sample feature matrices corresponding to the sample image to be identified, to obtain N encoded sample feature matrices, wherein the sample image to be identified includes a plurality of target identification objects, the sample image to be identified corresponds to label information, the resolutions of the M first sample feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer;

[0053] A second enhancement module is used to perform feature enhancement processing on the N coded sample feature matrices to obtain N enhanced sample feature matrices;

[0054] A second decoding module is used to decode the N enhanced sample feature matrices to obtain a binary image with the same resolution as the image to be identified; and

[0055] The model optimization module is used to train a deep learning model using the above label information and the above sample target extraction results to obtain a target extraction model.

[0056] According to the target extraction method based on remote sensing images provided by the present disclosure, N rounds of processing are performed on the M first feature matrices corresponding to the image to be identified to obtain N coding feature matrices. Since the resolutions of the M first feature matrices are different from each other, the features of target identification objects with different resolutions can be fused during the N rounds of processing of the M first feature matrices, thereby improving the accuracy of the coding feature matrices. The N coding feature matrices are subjected to feature enhancement processing. Since the coding feature matrix with a large resolution contains more features of the target identification object, more features of the target identification object can be retained by increasing the weight of the coding feature matrix with a large resolution, thereby improving the accuracy of the enhanced feature matrix. The target extraction result is determined by using the number of pixels of the target identification object in the binary image obtained after decoding the N enhanced feature matrices and the spatial resolution of the image to be identified, thereby improving the accuracy of the target extraction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0058] Figure 1 The application scenario diagram of target extraction according to the embodiment of the present disclosure is schematically shown;

[0059] Figure 2 A flowchart of a target extraction method according to an embodiment of the present disclosure is schematically shown;

[0060] Figure 3 A schematic diagram schematically shows a feature fusion process according to an embodiment of the present disclosure;

[0061] Figure 4 A schematic diagram schematically shows a context enhancement process according to an embodiment of the present disclosure;

[0062] Figure 5 A schematic diagram schematically shows feature enhancement processing of a coding feature matrix according to an embodiment of the present disclosure;

[0063] Figure 6 A flowchart of a method for training a target extraction model according to an embodiment of the present disclosure is schematically shown;

[0064] Figure 7 A schematic diagram schematically shows a comparison between a sample image to be identified and a sample binary image according to an embodiment of the present disclosure;

[0065] Figure 8 The schematic diagram of the structure of the target extraction model according to the embodiment of the present disclosure is shown;

[0066] Fig. 9A structural block diagram of a target extraction device according to an embodiment of the present disclosure is schematically shown; and

[0067] Fig.10 A structural block diagram of the training of the target extraction model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0068] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0069] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0070] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0071] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0072] In the technical solution disclosed in the present invention, the data involved, such as images to be identified, are all data that have been fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, and necessary confidentiality measures are taken without violating public order and good morals.

[0073] Sheep are the most numerous and representative livestock in animal husbandry. Sheep flocks are widely distributed in various regions. Accurately determining the location, distribution and size of sheep flocks can provide data support for the statistics of animal husbandry economy, and help management departments better plan and manage animal husbandry behavior and prevent grassland degradation caused by overgrazing.

[0074] With the continuous development of high spatial resolution remote sensing technology and the rapid iteration of deep learning technology, more and more high-resolution satellite image data are being applied to target monitoring tasks in various fields, and fruitful results have been achieved. In the sheep monitoring and sheep extraction tasks based on remote sensing images, since most sheep are white in remote sensing images, which is obviously different from the background color, the color threshold segmentation method is usually used to extract the distribution of sheep in remote sensing images. However, in scenes with complex backgrounds, the color threshold segmentation method is difficult to distinguish other backgrounds in the background that have similar spectral characteristics and scale characteristics as the target identification object, such as stones, bare land and other background objects in the background.

[0075] In addition, computer vision algorithms are often combined in this field to extract the distribution of sheep in remote sensing images. For example, support vector machines are used in feature extraction and classification algorithms to distinguish between sheep and image backgrounds. However, even in combination with computer vision algorithms, when the contrast between the color of the target object and the background color is low, some target objects cannot be extracted. In addition, when the volume of the target object is small, due to the small size of the target object, for example, a small sheep only occupies 4-6 pixels in a remote sensing image with a spatial resolution of 0.5 meters. When the contrast between the target object and the background is low, common models or algorithms are difficult to capture sufficient features of small-sized target objects.

[0076] Based on the above problems, an embodiment of the present disclosure provides a target extraction method, including: performing N rounds of processing on M first feature matrices corresponding to an image to be identified to obtain N coding feature matrices, wherein the image to be identified includes multiple target identification objects, and the resolution sizes of the M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer; performing feature enhancement processing on the N coding feature matrices to obtain N enhanced feature matrices; performing decoding processing on the N enhanced feature matrices to obtain a binary image with the same resolution size as the image to be identified; and determining a target extraction result based on the number of pixels of the target identification object in the binary image and the spatial resolution of the image to be identified, wherein the target extraction result represents the number of target identification objects.

[0077] Figure 1 The application scenario diagram of target extraction according to an embodiment of the present disclosure is schematically shown.

[0078] like Figure 1As shown, the application scenario 100 according to this embodiment may include an image acquisition device 101, a data receiving device 102, a network 103 and a server 104. The network 103 is used to provide a medium for a communication link between the data receiving device 102 and the server 104. The network 103 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0079] The image acquisition device 101 is used to acquire the image to be identified, which may refer to a remote sensing image. The image acquisition device 101 may include: Beijing-3, Sustainable Development Goals Satellite-1 (SDGSAT-1), and resource satellites. The data receiving device 102 is used to receive the remote sensing image acquired by the image acquisition device 101. The data receiving device 102 may include a ground base station, etc.

[0080] The server 104 may be a server that provides various services, for example, processing and analyzing the image to be recognized received through the network 103. It should be noted that the target extraction method and the target extraction model training method provided in the embodiment of the present disclosure can generally be executed by the server 104. Accordingly, the target extraction device and the target extraction model training device provided in the embodiment of the present disclosure can generally be set in the server 104. Accordingly, the target extraction device and the target extraction model training device provided in the embodiment of the present disclosure can also be set in a server different from the server 104 or a server cluster.

[0081] It should be understood that Figure 1 The number of image acquisition devices, data receiving devices, networks and servers in the embodiment is only for illustration. Any number of image acquisition devices, data receiving devices, networks and servers may be provided according to the implementation requirements.

[0082] The following will be based on Figure 1 The scene described by Figure 2 to Figure 5 The target extraction method of the disclosed embodiment is described in detail.

[0083] Figure 2 The flowchart of the target extraction method according to the embodiment of the present disclosure is schematically shown.

[0084] like Figure 2 As shown, the target extraction method of this embodiment includes operations S210 to S240.

[0085] In operation S210, N rounds of processing are performed on M first feature matrices corresponding to the image to be recognized to obtain N encoding feature matrices.

[0086] According to an embodiment of the present disclosure, the image to be identified includes multiple target identification objects, the resolution sizes of M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer.

[0087] In operation S220, feature enhancement processing is performed on the N encoding feature matrices to obtain N enhanced feature matrices.

[0088] In operation S230, the N enhanced feature matrices are decoded to obtain a binary image having the same resolution as the image to be recognized.

[0089] In operation S240, a target extraction result is determined based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized.

[0090] According to an embodiment of the present disclosure, the target extraction result characterizes the number of target recognition objects. For example, the target recognition object may refer to a sheep. According to an embodiment of the present disclosure, the image to be recognized may be used to characterize a remote sensing image collected from a satellite through visual interpretation. In order to ensure the accuracy of recognition during the recognition process, each remote sensing image may be cropped into an image block of a preset pixel size. For example, the preset pixel size may be 512×512. In the process of encoding the image to be recognized to obtain the first feature matrix, the image channels may be selected from three channels: green, blue, and near infrared.

[0091] According to an embodiment of the present disclosure, a self-attention and orientation encoding network (SOE-Net) is used as the basis for implementing the target extraction method. The self-attention and orientation encoding network adopts an encoder-decoder structure, and the image to be identified is encoded into a first feature matrix with the same resolution as the image to be identified through the encoder. Based on upsampling and downsampling operations on the first feature matrix, (M-1) first feature matrices with different resolutions are obtained. For example, in the case of M=3, the first feature matrix of 3×512×512 is upsampled to obtain a first feature matrix of 3×1024×1024, and the first feature matrix of 3×512×512 is downsampled to obtain a first feature matrix of 3×256×256.

[0092] According to an embodiment of the present disclosure, N coding feature matrices are obtained by performing N rounds of processing on M first feature matrices, wherein one coding feature matrix is ​​obtained after each round of processing. Feature enhancement processing is performed on the coding feature matrices obtained from each round of processing to obtain enhanced feature matrices. Since there are more features of target recognition objects in the feature map with a large resolution, a larger weight is given to the feature map with a large resolution through feature enhancement processing, thereby retaining more features of the target recognition object.

[0093] According to the embodiments of the present disclosure, the N enhanced feature matrices after feature enhancement processing are decoded to obtain a binary image with the same resolution as the image to be identified input to SOE-Net. The binary image output by SOE-Net contains only two colors, black and white, and the grayscale information is removed, which can more prominently display the target identification object, reduce interference factors, and facilitate subsequent target detection and segmentation. The binary image output by SOE-Net is kept at the same resolution as the input image to be identified, so as to facilitate the subsequent determination of the actual area occupied by the target identification object in the binary image according to the spatial resolution of the image to be identified.

[0094] According to an embodiment of the present disclosure, by identifying the number of pixels occupied by the target identification object in the binary image, since the resolution of the binary image and the image to be identified are the same, the area occupied by the target identification object can be calculated by the spatial resolution of the image to be identified and the number of pixels occupied by the target identification object, and the number of target identification objects can be estimated based on the average density of the target identification objects.

[0095] According to an embodiment of the present disclosure, N rounds of processing are performed on M first feature matrices corresponding to the image to be identified to obtain N coding feature matrices. Since the resolutions of the M first feature matrices are different from each other, the features of target identification objects with different resolutions can be fused during the N rounds of processing of the M first feature matrices, thereby improving the accuracy of the coding feature matrices. Feature enhancement processing is performed on the N coding feature matrices. Since the coding feature matrix with a large resolution contains more features of the target identification object, more features of the target identification object can be retained by increasing the weight of the coding feature matrix with a large resolution, thereby improving the accuracy of the enhanced feature matrix. The target extraction result is determined by using the number of pixels of the target identification object in the binary image obtained after decoding the N enhanced feature matrices and the spatial resolution of the image to be identified, thereby improving the accuracy of the target extraction result.

[0096] According to an embodiment of the present disclosure, N rounds of processing are performed on M first feature matrices corresponding to the image to be identified to obtain N encoding feature matrices, including:

[0097] When n = 1, perform feature fusion processing on each of the M first feature matrices in the first round to obtain M fused feature matrices in the first round; perform context enhancement processing on each of the M first feature matrices in the first round to obtain M context-enhanced matrices in the first round; fuse the M fused feature matrices and the M context-enhanced matrices to obtain M second feature matrices in the second round; when 1 < n ≤ N, perform feature fusion processing on each of the M second feature matrices in the (n - 1)th round to obtain M fused feature matrices in the (n - 1)th round; perform context enhancement processing on each of the M second feature matrices in the (n - 1)th round to obtain M context-enhanced matrices in the (n - 1)th round; fuse each of the M fused feature matrices and the context-enhanced matrix corresponding to each of the M fused feature matrices to obtain M second feature matrices in the nth round; perform resampling operations on the M second feature matrices in the nth round to obtain M resampled feature matrices; and fuse the M resampled feature matrices to obtain the encoded feature matrix in the (n - 1)th round.

[0098] According to an embodiment of the present disclosure, n represents the number of rounds of processing on the M first feature matrices. By performing feature fusion processing on the M first feature matrices, the M feature matrices in the first round are fused with each other to obtain M fused feature matrices. By performing context enhancement processing on the M first feature matrices, it helps the target extraction model learn the spatial position relationship between the target recognition object and the surrounding environment to obtain M context-enhanced matrices in the first round. The fusion methods of the feature matrices include two methods: concat and add. The matrix fusion in the present disclosure can be completed by the add method.

[0099] According to an embodiment of the present disclosure, fuse the M fused feature matrices and the M context-enhanced matrices in the first round. Among them, each first feature matrix has a corresponding fused feature matrix and a context-enhanced matrix. Fuse the fused feature matrix and the context-enhanced matrix corresponding to each first feature matrix respectively to obtain M second feature matrices corresponding to each of the M first feature matrices.

[0100] According to an embodiment of the present disclosure, perform feature fusion processing on each of the M second feature matrices in the nth round to obtain M fused feature matrices in the nth round; perform context enhancement processing on each of the M second feature matrices in the nth round to obtain M context-enhanced matrices in the nth round; fuse each of the M fused feature matrices and the context-enhanced matrix corresponding to each of the M fused feature matrices to obtain M second feature matrices; perform resampling operations on the M second feature matrices to obtain M resampled feature matrices; and fuse the M resampled feature matrices to obtain the encoded feature matrix in the nth round.

[0101] According to an embodiment of the present disclosure, the second feature matrix obtained in the previous round of processing is used as the matrix for the next round of processing, and the image feature matrices with different resolutions are processed multiple times to reduce the influence of noise in the feature matrix on the target extraction result. The N encoding feature matrices with different resolutions and number of channels obtained by each round of processing can retain the features of the image to be identified as much as possible, because some features may be easier to extract in a low-resolution image, while other features may be more significant in a high-resolution image, so that the N encoding feature matrices can be used to retain the features of the image to be identified as much as possible, so as to obtain more abundant features of the image to be identified in subsequent processing, thereby improving the accuracy of the target extraction result.

[0102] According to an embodiment of the present disclosure, feature fusion processing is performed on the M second feature matrices of the (n-1)th round respectively to obtain the M fused feature matrices of the (n-1)th round, including:

[0103] The i-th second feature matrix of the (n-1)th round is convolved to obtain the i-th convolution feature matrix, where i is a positive integer less than or equal to M; based on the (M-1) second feature matrix of the (n-1)th round, the (M-1) feature matrices to be fused are determined, where the resolution of the (M-1) feature matrices to be fused is the same as that of the i-th convolution feature matrix; and the i-th convolution feature matrix is ​​fused with the (M-1) feature matrices to be fused to obtain the i-th fused feature matrix.

[0104] According to an embodiment of the present disclosure, convolution processing can extract and process detail information in the second feature matrix, thereby obtaining a richer and more representative feature representation. The convolution feature matrix represents the feature matrix after the convolution operation is performed on the second feature matrix. The feature matrix to be fused represents the matrix used to fuse with the i-th second feature matrix of the (n-1)th round. The feature matrix to be fused is obtained by resampling any one of the (M-1) second feature matrices, wherein the (M-1) second feature matrices are the second feature matrices of the other (n-1)th rounds except the i-th second feature matrix.

[0105] Figure 3 A schematic diagram of a feature fusion process according to an embodiment of the present disclosure is schematically shown.

[0106] like Figure 3As shown, taking M = 3 as an example, when n = 1, Input 1, Input 2, and Input 3 respectively represent the first feature matrices with three different resolutions. When 1 < n ≤ N, Input 1, Input 2, and Input 3 respectively represent the second feature matrices with three different resolutions in the (n - 1)-th round. The resolution of the first feature matrix or the second feature matrix represented by Input 1 is greater than the resolutions of the first feature matrix or the second feature matrices represented by Input 2 and Input 3. The resolution of the first feature matrix or the second feature matrix represented by Input 2 is greater than the resolution of the first feature matrix or the second feature matrix represented by Input 3. Output 1, Output 2, and Output 3 respectively represent the fused feature matrices with three different resolutions in the (n - 1)-th round or the n-th round.

[0107] For Input 1, after a 3×3 convolution operation on Input 1, the first convolution feature matrix is obtained, and downsampling operations are respectively performed on the first convolution feature matrix to obtain the first to-be-fused feature matrix with the same resolution as Input 2 and the second to-be-fused feature matrix with the same resolution as Input 3.

[0108] For Input 2, Input 2 is fused with the first to-be-fused feature matrix, and a 3×3 convolution kernel is used to perform a convolution operation on the fused matrix to obtain the second convolution feature matrix. The second convolution feature matrix is upsampled to obtain the third to-be-fused feature matrix with the same resolution as Input 1. The feature matrix obtained after the convolution operation is downsampled to obtain the fourth to-be-fused feature matrix with the same resolution as Input 3.

[0109] For Input 3, Input 3 is fused with the second to-be-fused feature matrix and the fourth to-be-fused feature matrix, and a 3×3 convolution kernel is used to perform a convolution operation on the fused matrix to obtain Output 3, that is, the fused feature matrix corresponding to Input 3. Upsampling operations are respectively performed on the obtained Output 3 to obtain the fifth to-be-fused feature matrix with the same resolution as Input 1 and the sixth to-be-fused feature matrix with the same resolution as Input 2.

[0110] The first convolution feature matrix is fused with the third to-be-fused feature matrix and the fifth to-be-fused feature matrix to obtain Output 1, that is, the fused feature matrix corresponding to Input 1. The second convolution feature matrix is fused with the first to-be-fused feature matrix and the sixth to-be-fused feature matrix to obtain Output 2, that is, the fused feature matrix corresponding to Input 2.

[0111] According to an embodiment of the present disclosure, by first performing a convolution operation on a first feature matrix or a second feature matrix with a higher resolution, its features can be mapped to a lower dimensional space, and then fused with other first feature matrices or second feature matrices. Such an operation helps to establish a connection between first feature matrices or second feature matrices with different resolutions, thereby better integrating the information of the M first feature matrices or the M second feature matrices.

[0112] According to an embodiment of the present disclosure, context enhancement processing is performed on the M second feature matrices of the (n-1)th round respectively to obtain the M context enhancement matrices of the (n-1)th round, including:

[0113] The M second feature matrices of the (n-1)th round are feature fused to obtain the (n-1)th round of context enhancement matrix, wherein the number of channels of the (n-1)th round of context enhancement matrix is ​​the sum of the number of channels of the M second feature matrices of the (n-1)th round; the (n-1)th round of context enhancement matrix is ​​input into the (n-1)th round of channel attention module to obtain the (n-1)th round of channel enhancement feature matrix, wherein the (n-1)th round of channel attention module is used to adjust the weights of multiple channels of the (n-1)th round of context enhancement matrix; and the (n-1)th round of channel enhancement feature matrix is ​​resampled to obtain M context enhancement matrices, wherein the matrix size of the (n-1)th round of M context enhancement matrices corresponds one-to-one to the resolution size of the (n-1)th round of M fused feature matrices.

[0114] According to an embodiment of the present disclosure, during the context enhancement process of the M second feature matrices or the M first feature matrices, the channel attention module may be an attention mechanism (Squeeze and Extraction, SE). The matrix to be context enhanced is obtained by feature fusion of the second feature matrix, or by feature fusion of the first feature matrix. Since the resolutions of the M second feature matrices or the M first feature matrices are different, before fusing the M second feature matrices, the M feature matrices need to be resampled to transform the M second feature matrices or the M first feature matrices into matrices with the same resolution.

[0115] According to an embodiment of the present disclosure, the number of channels of the matrix to be context-enhanced is the sum of the number of channels of the M second feature matrices, or the sum of the number of channels of the M first feature matrices. The matrix to be context-enhanced is input into the channel attention module, and a channel-enhanced feature matrix is ​​output. To facilitate fusion with the fused feature matrix, the channel-enhanced feature matrix is ​​resampled so that the resolution sizes of the M context-enhanced matrices correspond one-to-one to the resolution sizes of the M fused feature matrices.

[0116] Figure 4A schematic diagram showing a context enhancement process according to an embodiment of the present disclosure is schematically illustrated.

[0117] As Figure 4 shown, taking M = 3 as an example, when n = 1, Input 1, Input 2, and Input 3 respectively represent the first feature matrices of three different resolutions. When 1 < n ≤ N, Input 1, Input 2, and Input 3 respectively represent the second feature matrices of three different resolutions in the (n - 1)th round. The resolution of the first feature matrix or the second feature matrix represented by Input 1 is greater than the resolution of the first feature matrix or the second feature matrix represented by Input 2 and Input 3. The resolution of the first feature matrix or the second feature matrix represented by Input 2 is greater than the resolution of the first feature matrix or the second feature matrix represented by Input 3.

[0118] Output a, Output b, and Output c respectively represent the context enhancement matrices of three different resolutions in the (n - 1)th round or the nth round. C represents the number of channels of the matrix to be context-enhanced, and H×W represents the resolution size of the context enhancement matrix.

[0119] The Input 1, Input 2, and Input 3 are subjected to feature fusion to obtain the matrix to be context-enhanced. Among them, the number of channels of the matrix to be context-enhanced is the sum of the number of channels of the M second feature matrices in the (n - 1)th round. Input 1 is downsampled or pooled and then converted into a feature matrix with the same resolution size as Input 2, and Input 3 is upsampled and then converted into a feature matrix with the same resolution size as Input 2. Input 2 is fused with the converted Input 1 and Input 3 to obtain a matrix to be context-enhanced with a size of C×H×W. Among them, C is the sum of the number of channels of Input 1, Input 2, and Input 3, and H×W is the resolution size of Input 2.

[0120] After passing the matrix to be context-enhanced through the channel attention module, a channel-enhanced feature matrix with a size of C×H×W is obtained. The channel-enhanced feature matrix is upsampled to obtain Output a with the same resolution size as Input 1. The channel-enhanced feature matrix is upsampled to obtain Output c with the same resolution size as Input 3. The matrix size of Output b is C×H×W.

[0121] According to an embodiment of the present disclosure, by performing context enhancement processing on M second feature matrices or M first feature matrices, the weights of each channel in the M second feature matrices or M first feature matrices are adaptively adjusted, so that the target extraction model pays more attention to the feature information of the target recognition object. Through the SE module, by adaptively weighting each channel, the model can pay more attention to the important context information in the second feature matrix or the first feature matrix, thereby improving the target extraction model's ability to understand the overall structure of the image, and thus improving the accuracy of recognizing the to-be-recognized image including complex scenes and multiple objects. In addition, adaptively weighting the channels of multiple second feature matrices or multiple first feature matrices can help the context enhancement matrix learn the channel information in different second feature matrices or first feature matrices, reduce the misclassification rate, that is, improve the accuracy of the binary image, and thus improve the accuracy of the target extraction result.

[0122] According to an embodiment of the present disclosure, performing feature enhancement processing on N encoded feature matrices to obtain N enhanced feature matrices, including:

[0123] Performing downsampling operations on the first (j - 1) encoded feature matrices to obtain (j - 1) downsampled feature matrices; fusing the (j - 1) downsampled feature matrices with the jth encoded feature matrix to obtain the jth initial enhanced feature matrix, where the resolution of the jth encoded feature matrix is less than the resolution of the first (j - 1) encoded feature matrices, 1 < j ≤ N and j is an integer; and inputting the jth initial enhanced feature matrix into the enhanced channel attention module corresponding to the jth encoded feature matrix to obtain the jth enhanced feature matrix, where the enhanced channel attention module corresponding to the jth encoded feature matrix is used to adjust the weights of multiple channels of the jth initial enhanced feature matrix.

[0124] According to an embodiment of the present disclosure, the resolution sizes and the number of channels of the N encoded feature matrices may all be different. In the complete processing process, the resolution of the (j - 1)th encoded feature matrix is greater than the resolution of the (j - 1)th encoded feature matrix. Performing downsampling operations on the first (j - 1) encoded feature matrices, and fusing the obtained (j - 1) downsampled feature matrices with the jth encoded feature matrix to obtain the jth initial enhanced feature matrix. During the fusion process, the number of channels of the initial enhanced feature matrix can be made the same as that of the encoded feature matrix through linear transformation or convolution operations.

[0125] According to an embodiment of the present disclosure, the enhanced channel attention module represents the modified SE module. Since the number of channels and resolution size of each encoding feature matrix are different, a corresponding enhanced channel attention module is set for each initial enhanced feature matrix to match each initial enhanced feature matrix. The enhanced channel attention module of the present disclosure multiplies the encoding feature matrix and the channel attention information matrix obtained by spatial feature compression and channel feature learning of the initial enhanced feature matrix by the weight coefficient channel by channel, and finally outputs an enhanced feature matrix with channel attention.

[0126] Figure 5 A schematic diagram of performing feature enhancement processing on a coding feature matrix according to an embodiment of the present disclosure is schematically shown.

[0127] like Figure 5 As shown in the figure, the resolution and number of channels of encoding feature matrix 1, encoding feature matrix 2 and encoding feature matrix 3 are different. It is assumed that the matrix size of encoding feature matrix 1 is 16×256×256, the matrix size of encoding feature matrix 2 is 32×128×128, and the matrix size of encoding feature matrix 3 is 64×64×64. The resolution of encoding feature matrix 1 is greater than that of encoding feature matrix 2 and encoding feature matrix 3, and the resolution of encoding feature matrix 2 is greater than that of encoding feature matrix 3. The enhanced SE module is the enhanced channel attention module.

[0128] For the encoding feature matrix 1, the encoding feature matrix 1 is downsampled to obtain the first downsampled feature matrix with the same resolution as the encoding feature matrix 2, and the second downsampled feature matrix with the same resolution as the encoding feature matrix 3. The encoding feature matrix 1 is input into the corresponding enhanced SE module, where the enhanced SE module performs spatial feature compression and channel feature learning on the encoding feature matrix 1 to obtain a 16×1×1 channel attention information matrix. The 16×256×256 encoding feature matrix 1 and the 16×1×1 channel attention information matrix are multiplied channel by channel by the weight coefficient to obtain the enhanced feature matrix 1 with channel attention.

[0129] For the encoding feature matrix 2, the encoding feature matrix 2 is downsampled to obtain the third downsampled feature matrix with the same resolution as the encoding feature matrix 3. The first downsampled feature matrix is ​​fused with the encoding feature matrix 2 to obtain the initial enhanced feature matrix, and the initial enhanced feature matrix is ​​input into the corresponding enhanced SE module, wherein the enhanced SE module performs spatial feature compression and channel feature learning on the initial enhanced feature matrix to obtain a 32×1×1 channel attention information matrix, and the 32×128×128 encoding feature matrix 2 and the 32×1×1 channel attention information matrix are multiplied channel by channel by the weight coefficient to obtain the enhanced feature matrix 2 with channel attention.

[0130] The second down-sampling feature matrix and the third down-sampling feature matrix are fused with the encoding feature matrix 3 to obtain the initial enhanced feature matrix, and the initial enhanced feature matrix is ​​input into the corresponding enhanced SE module, wherein the enhanced SE module performs spatial feature compression and channel feature learning on the initial enhanced feature matrix to obtain a 64×1×1 channel attention information matrix, and the 64×64×64 encoding feature matrix 3 and the 64×1×1 channel attention information matrix are multiplied channel by channel by the weight coefficient to obtain the enhanced feature matrix 3 with channel attention.

[0131] According to an embodiment of the present disclosure, in the presence of coding feature matrices of different resolution sizes and numbers of channels, the coding feature matrix with higher resolution is downsampled and fused with the coding feature matrix with lower resolution, and the fused initial enhanced feature matrix is ​​input into the enhanced channel attention module. Since the coding feature matrix with higher resolution contains more features of the target recognition object, the high-resolution coding feature matrix is ​​downsampled, and its feature representation can be converted into a coarser but more global representation, so as to better align with the coding feature matrix with lower resolution in spatial range, and at the same time, the information in the high-resolution and low-resolution coding feature matrices is used to obtain a more comprehensive and rich feature representation, thereby improving the accuracy of the final target extraction result. In addition, the fused initial enhanced feature matrix is ​​input into the enhanced channel attention module, and the features of the target recognition object in the fused feature matrix can be enhanced by the enhanced channel attention module to further improve the accuracy of target extraction.

[0132] According to an embodiment of the present disclosure, the encoded feature matrix and the matrix obtained by performing spatial feature compression and channel feature learning on the initial enhanced feature matrix are multiplied channel by channel by weight coefficients to finally obtain an enhanced feature matrix. This can give channels of the large-resolution encoded feature matrix greater weights while retaining more features of the target recognition object.

[0133] According to an embodiment of the present disclosure, N enhanced feature matrices are decoded to obtain a binary image having the same resolution as the image to be identified, including:

[0134] The multi-scale feature fusion technology is used to fuse N enhanced feature matrices to obtain a multi-scale fusion matrix; the multi-scale fusion matrix is ​​resampled to obtain a reconstructed feature matrix; the reconstructed feature matrix is ​​processed using an activation function to obtain a threshold feature matrix, wherein each pixel value in the threshold feature matrix is ​​between 0 and 1; the threshold feature matrix is ​​binarized according to a preset pixel threshold to obtain a binary image.

[0135] According to an embodiment of the present disclosure, a multi-scale feature fusion technology is used to fuse matrices of different scales. The multi-scale feature fusion technology may include pyramid pooling, etc., which is used to fuse feature information from different scales to generate a more expressive feature matrix. The N enhanced feature matrices are pooled separately, and methods such as average pooling or maximum pooling can be used to cascade each pooled feature map together to obtain a scale fusion matrix.

[0136] According to an embodiment of the present disclosure, in order to ensure that the binary image is the same size as the image to be identified, the multi-scale fusion matrix is ​​resampled to obtain a reconstructed feature matrix. After being processed by the activation function, the pixel values ​​in the obtained threshold feature matrix may be in different ranges. According to a preset pixel threshold, the pixel values ​​in the threshold feature matrix are converted into binary values.

[0137] According to an embodiment of the present disclosure, the preset pixel threshold represents a threshold for dividing pixel values ​​between 0 and 1 into 0 or 1, thereby distinguishing the target recognition object from the background. Pixels greater than the preset pixel threshold are set to 1, and pixels less than the threshold are set to 0, thereby determining a binary image. Pixels with a pixel value of 0 can be used to represent the target recognition object, and pixels with a pixel value of 1 can be used to represent the background.

[0138] According to the implementation of the present disclosure, the fusion of enhanced feature matrices of multiple scales can enhance the expression ability of the multi-scale fusion matrix and improve the accuracy of the binary image. In addition, the reconstructed feature matrix is ​​processed using an activation function so that the pixel values ​​of the reconstructed feature matrix are between 0 and 1 to obtain a threshold feature matrix, and the threshold feature matrix is ​​binarized according to a preset pixel threshold to obtain a binary image, so as to reduce the influence of noise and unnecessary details in the threshold feature matrix on the target extraction result and improve the accuracy of the target extraction result.

[0139] According to an embodiment of the present disclosure, based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized, determining the target extraction result includes:

[0140] The binary image is processed by using an image segmentation algorithm to obtain the number of pixels of the target recognition object in the binary image; the area occupied by the target recognition object is determined based on the spatial resolution and the number of pixels of the image to be recognized; and the number of target recognition objects is determined based on the area occupied and the average density of the target recognition objects.

[0141] According to an embodiment of the present disclosure, an image segmentation algorithm is used to separate the target object from the background. The image segmentation algorithm may include a global threshold method, an adaptive threshold method, an Otsu's binarization algorithm, etc. The target object is separated from the background by the image segmentation algorithm, and the number of pixels of the segmented image is counted to obtain the number of pixels of the target object. Assuming that the spatial resolution of the image to be identified is R, the area R of each pixel is used. 2 Multiply by the number of pixels to get the total area. Multiply the area of ​​the target objects by the estimated average density of the target objects to get the number of target objects.

[0142] According to the embodiments of the present disclosure, the actual area occupied by the target recognition object can be more accurately estimated by combining the number of pixels of the target recognition object in the binary image with the spatial resolution of the image to be recognized, and then the number of target recognition objects can be inferred. This estimation method can more accurately determine the number of target recognition objects when the number of pixels of the image to be recognized is low, thereby improving the accuracy of the target extraction results.

[0143] Figure 6 A flowchart of a method for training a target extraction model according to an embodiment of the present disclosure is schematically shown.

[0144] like Figure 6 As shown, the target extraction method of this embodiment includes operations S610 to S640:

[0145] In operation S610, N rounds of processing are performed on M first sample feature matrices corresponding to the sample image to be identified to obtain N encoded sample feature matrices.

[0146] According to an embodiment of the present disclosure, the sample image to be identified includes multiple target identification objects, the sample image to be identified corresponds to label information, the resolution sizes of M first sample feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer.

[0147] In operation S620, feature enhancement processing is performed on the N encoded sample feature matrices to obtain N enhanced sample feature matrices.

[0148] In operation S630, the N enhanced sample feature matrices are decoded to obtain a sample binary image having the same resolution as the image to be identified.

[0149] In operation S640, a deep learning model is trained using the label information and the sample binary image to obtain an object extraction model.

[0150] According to an embodiment of the present disclosure, the target extraction model is implemented based on SOE-Net. During the training of the target extraction model, when the target recognition object is a sheep, the sample image to be recognized can select a 512×512 resolution image with a scale size of 3 channels. The deep learning model processes the M first sample feature matrices corresponding to the sample image to be recognized for N rounds to obtain N encoding feature matrices. The label information corresponding to the sample image to be recognized is used to characterize the actual number of pixels of the target recognition object in the sample image to be recognized.

[0151] According to an embodiment of the present disclosure, N encoding sample feature matrices are obtained by performing N rounds of processing on M first sample feature matrices, wherein one encoding sample feature matrix is ​​obtained after each round of processing. Feature enhancement processing is performed on the encoding sample feature matrix obtained after each round of processing. Since there are more sheep features in the feature map with a large resolution, more features of the target recognition object are retained by giving the feature map with a large resolution a larger weight.

[0152] According to the embodiments of the present disclosure, the N enhanced sample feature matrices after feature enhancement processing are decoded to obtain a binary image with the same resolution as the sample image to be identified that is input to the target extraction model. The output sample binary image of the target extraction model contains only two colors, black and white, and the grayscale information is removed, which can more prominently display the target identification object, reduce interference factors, and facilitate the subsequent detection and segmentation of the target identification object. The sample binary image output by the target extraction model is kept at the same resolution as the input sample image to be identified, so as to facilitate the subsequent determination of the actual area occupied by the target identification object in the sample binary image according to the spatial resolution of the sample image to be identified.

[0153] According to the embodiments of the present disclosure, during the implementation process, the learning rate of the target extraction model is preferably set to 0.001, and the optimization strategy preferably adopts stochastic gradient descent (SGD). The trained target extraction model is tested in the test set to obtain the target extraction result of the test set, that is, the number of sheep, wherein the test set includes sample images to be identified and sample labels that are not used for model training.

[0154] Figure 7 The following is a schematic diagram showing the comparison between a sample image to be identified and a sample binary image according to an embodiment of the present disclosure.

[0155] like Figure 7As shown, 701, 702 and 703 represent three sample images to be identified, and 704, 705 and 706 represent sample binary images corresponding to 701, 702 and 703, respectively. From the comparison between the sample images to be identified and the sample binary images, it can be seen that when there are many background objects with various sizes, the target extraction model proposed in the present disclosure can better solve the problem of background interference and accurately extract the target identification object. Even if the background interference objects have a high similarity with sheep, the trained target extraction model can still successfully extract most of the target identification objects while avoiding interference.

[0156] According to an embodiment of the present disclosure, the accuracy of the target extraction model is evaluated by calculating the intersection over union (IoU), F1 value, recall rate (Recall) and precision (Precision) of the target extraction model, and the statistical results shown in Table 1, which is a statistical table of target recognition object extraction accuracy verification, are obtained.

[0157] Table 1 Statistical table of target recognition object extraction accuracy verification

[0158] IoU Recall Precision F1 71.5％ 58.9％ 81.8％ 70.4％

[0159] According to an embodiment of the present disclosure, the IOU can be calculated by the following formula (1):

[0160]

[0161] Among them, TP is the number of pixels extracted as target recognition objects by the target extraction model; TN is the number of real pixels extracted as background by the target extraction model; FP is the number of pixels of the background that are mistakenly identified as target recognition objects by the target extraction model.

[0162] According to an embodiment of the present disclosure, Precision can be calculated by the following formula (2):

[0163]

[0164] According to an embodiment of the present disclosure, Recall can be calculated by the following formula (3):

[0165]

[0166] Among them, FN is the number of pixels of the target recognition object that is mistakenly identified as background by the target extraction model.

[0167] According to an embodiment of the present disclosure, the F1 value can be calculated by the following formula (4):

[0168]

[0169] Figure 8 The structural diagram of the target extraction model according to the embodiment of the present disclosure is schematically shown.

[0170] like Figure 8 As shown in the figure, when the target extraction model is used to identify the number of sheep, the 3×512×512 image to be identified is used as the input of the target extraction model. The first encoding module of the target extraction model is used to encode the image to be identified to obtain a 3×512×512 first feature matrix, the 3×512×512 first feature matrix is ​​upsampled to obtain a 3×1024×1024 first feature matrix, and the 3×512×512 first feature matrix is ​​downsampled to obtain a 3×256×256 first feature matrix. The first fusion submodule has the same function as the second fusion submodule, and the first enhancement submodule has the same function as the second enhancement submodule, where "first" and "second" are only used to distinguish the processing of the first round from that of other rounds.

[0171] The first fusion submodule is used to perform feature fusion processing on the three first feature matrices respectively to obtain M fused feature matrices of the first round; the first enhancement submodule is used to perform context enhancement processing on the three first feature matrices respectively to obtain three context enhancement matrices of the first round. The three fused feature matrices and the context enhancement matrices corresponding to the three fused feature matrices are fused respectively to obtain three second feature matrices of the second round, namely, three matrices of 16×512×512, 16×256×256, and 16×128×228. The second feature matrix of 16×512×512 is downsampled to obtain a feature matrix of 16×256×256, and the second feature matrix of 16×128×128 is upsampled to obtain a feature matrix of 16×256×256. The obtained feature matrix of 16×256×256 is fused with the feature matrix of 16×256×256 to obtain the encoding feature matrix of the first round.

[0172] The second fusion submodule is used to perform feature fusion processing on the three second feature matrices respectively to obtain the three fused feature matrices of the second round; the second enhancement submodule is used to perform context enhancement processing on the three second feature matrices respectively to obtain the three context enhancement matrices of the first round. The three fused feature matrices and the context enhancement matrices corresponding to the three fused feature matrices are fused respectively to obtain the three second feature matrices of the third round, namely, three matrices of 32×256×256, 32×128×228, and 32×64×64. Based on the same method as the first round, the encoding feature matrix of the second round is obtained. Based on the same method as the second round, the encoding feature matrix of the third round can be obtained.

[0173] The first feature enhancement module is used to perform feature enhancement processing on the three coded feature matrices to obtain three enhanced feature matrices. The first decoding module is used to decode the three enhanced feature matrices to obtain a binary image with the same resolution size as the image to be identified, that is, a binary image of size 2×512×512.

[0174] According to an embodiment of the present disclosure, in the process of sheep identification, taking into account the average number of pixels occupied by sheep on the image to be identified and the resolution of the input image to be identified, the resolution of the minimum second feature image in the first encoding module is controlled to 64×64, which can ensure that the final image resolution will not be too small, thereby reducing the computational complexity of the target extraction model on a matrix with a smaller resolution.

[0175] Based on the above target extraction method, the present disclosure also provides a target extraction device. Fig. 9 The device is described in detail.

[0176] Fig. 9 The structural block diagram of the target extraction device according to an embodiment of the present disclosure is schematically shown.

[0177] like Fig. 9 As shown, the target extraction device 900 of this embodiment includes a first encoding module 910 , a first enhancement module 920 , a first decoding module 930 and a result determination module 940 .

[0178] The first encoding module 910 is used to perform N rounds of processing on M first feature matrices corresponding to the image to be identified to obtain N encoding feature matrices, wherein the image to be identified includes multiple target identification objects, the resolution sizes of the M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer. In one embodiment, the first encoding module 910 can be used to perform the operation S210 described above, which will not be repeated here.

[0179] The first enhancement module 920 is used to perform feature enhancement processing on the N encoding feature matrices to obtain N enhanced feature matrices. In one embodiment, the first enhancement module 920 can be used to perform the operation S220 described above, which will not be described in detail here.

[0180] The first decoding module 930 is used to decode the N enhanced feature matrices to obtain a binary image with the same resolution as the image to be identified. In one embodiment, the first decoding module 930 can be used to perform the operation S230 described above, which will not be described in detail here.

[0181] A result determination module 940 is configured to determine a target extraction result based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized, where the target extraction result represents the number of the target recognition objects. In one embodiment, the result determination module 940 may be configured to perform the operation S240 described above, which will not be elaborated herein.

[0182] According to an embodiment of the present disclosure, the first encoding module 910 includes: a first fusion sub-module, a first enhancement sub-module, a first determination sub-module, a second fusion sub-module, a second enhancement sub-module, a second determination sub-module, a first sampling sub-module, and a third determination sub-module.

[0183] The first fusion sub-module is configured to, when n = 1, perform feature fusion processing on each of the M first feature matrices in the first round to obtain M fused feature matrices in the first round.

[0184] The first enhancement sub-module is configured to perform context enhancement processing on each of the M first feature matrices in the first round to obtain M context-enhanced matrices in the first round.

[0185] The first determination sub-module is configured to fuse the M fused feature matrices and the M context-enhanced matrices to obtain M second feature matrices in the second round.

[0186] The second fusion sub-module is configured to, when 1 < n ≤ N, perform feature fusion processing on each of the M second feature matrices in the (n - 1)th round to obtain M fused feature matrices in the (n - 1)th round.

[0187] The second enhancement sub-module is configured to perform context enhancement processing on each of the M second feature matrices in the (n - 1)th round to obtain M context-enhanced matrices in the (n - 1)th round.

[0188] The second determination sub-module is configured to fuse each of the M fused feature matrices and the context-enhanced matrix corresponding to each of the M fused feature matrices to obtain M second feature matrices in the nth round.

[0189] The first sampling sub-module is configured to perform resampling operations on the M second feature matrices in the nth round to obtain M resampled feature matrices.

[0190] The third determination sub-module is configured to fuse the M resampled feature matrices to obtain an encoded feature matrix in the (n - 1)th round.

[0191] According to an embodiment of the present disclosure, the second enhancement sub-module includes: a convolutional unit, a first determination unit, and a second determination unit.

[0192] A convolution unit is configured to perform convolution processing on the i-th second feature matrix in the (n - 1)-th round to obtain the i-th convolution feature matrix, where i is a positive integer less than or equal to M.

[0193] A first determination unit is configured to determine (M - 1) to-be-fused feature matrices based on the (M - 1) second feature matrices in the (n - 1)-th round, where the (M - 1) to-be-fused feature matrices have the same resolution size as the i-th convolution feature matrix. And

[0194] A second determination unit is configured to fuse the i-th convolution feature matrix with the (M - 1) to-be-fused feature matrices to obtain the i-th fused feature matrix.

[0195] According to an embodiment of the present disclosure, a second enhancement sub-module includes: a fusion unit, a third determination unit, and a resampling unit.

[0196] The fusion unit is configured to perform feature fusion on the M second feature matrices in the (n - 1)-th round to obtain a to-be-context-enhanced matrix in the (n - 1)-th round, where the number of channels of the to-be-context-enhanced matrix is the sum of the numbers of channels of the M second feature matrices in the (n - 1)-th round.

[0197] The third determination unit is configured to input the to-be-context-enhanced matrix in the (n - 1)-th round into the channel attention module in the (n - 1)-th round to obtain a channel-enhanced feature matrix in the (n - 1)-th round, where the channel attention module in the (n - 1)-th round is configured to adjust the weights of multiple channels of the to-be-context-enhanced matrix in the (n - 1)-th round.

[0198] The resampling unit is configured to perform resampling on the channel-enhanced feature matrix in the (n - 1)-th round to obtain M context-enhanced matrices, where the matrix sizes of the M context-enhanced matrices in the (n - 1)-th round correspond one-to-one to the resolution sizes of the M fused feature matrices in the (n - 1)-th round.

[0199] According to an embodiment of the present disclosure, a first enhancement module 920 includes: a downsampling sub-module, a fourth determination sub-module, and a fifth determination sub-module.

[0200] The downsampling sub-module is configured to perform downsampling operations on the first (j - 1) encoded feature matrices to obtain (j - 1) downsampled feature matrices.

[0201] The fourth determination sub-module is configured to fuse the (j - 1) downsampled feature matrices with the j-th encoded feature matrix to obtain the j-th initial enhancement feature matrix, where the resolution of the j-th encoded feature matrix is smaller than the resolutions of the first (j - 1) encoded feature matrices, 1 < j ≤ N and j is an integer.

[0202] The fifth determination submodule is used to input the jth initial enhanced feature matrix into the enhanced channel attention module corresponding to the jth encoding feature matrix to obtain the jth enhanced feature matrix, wherein the enhanced channel attention module corresponding to the jth encoding feature matrix is ​​used to adjust the weights of multiple channels of the jth initial enhanced feature matrix.

[0203] According to an embodiment of the present disclosure, the first decoding module 930 includes: a sixth determination submodule, a second sampling submodule, a seventh determination submodule, and an image determination submodule.

[0204] The sixth determination submodule is used to fuse the N enhanced feature matrices using a multi-scale feature fusion technology to obtain a multi-scale fusion matrix.

[0205] The second sampling submodule is used to resample the multi-scale fusion matrix to obtain a reconstructed feature matrix.

[0206] The seventh determination submodule is used to process the reconstructed feature matrix using an activation function to obtain a threshold feature matrix, wherein each pixel value in the threshold feature matrix is ​​between 0 and 1.

[0207] The image determination submodule is used to perform binarization processing on the threshold feature matrix according to a preset pixel threshold to obtain a binary image.

[0208] According to an embodiment of the present disclosure, the result determination module 940 includes: a pixel determination submodule, an area determination submodule, and an object determination submodule.

[0209] The pixel determination submodule is used to process the binary image using an image segmentation algorithm to obtain the number of pixels of the target recognition object in the binary image.

[0210] The area determination submodule is used to determine the area occupied by the target identification object based on the spatial resolution and the number of pixels of the image to be identified.

[0211] The object determination submodule is used to determine the number of target identification objects based on the occupied area and the average density of the target identification objects.

[0212] Based on the above-mentioned training method of the target extraction model, the present disclosure also provides a training device for the target extraction model. Fig.10 The device is described in detail.

[0213] Fig.10 A structural block diagram of the training of the target extraction model according to an embodiment of the present disclosure is schematically shown.

[0214] like Fig.10As shown, the training device 1000 of the target extraction model of this embodiment includes: a second encoding module 1010, a second enhancement module 1020, a second decoding module 1030, and a model optimization module 1040.

[0215] The second encoding module 1010 is used to perform N rounds of processing on M first sample feature matrices corresponding to the sample image to be identified, to obtain N encoded sample feature matrices, wherein the sample image to be identified includes a plurality of target identification objects, the sample image to be identified corresponds to label information, the resolutions of the M first sample feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer;

[0216] A second enhancement module 1020 is used to perform feature enhancement processing on the N coded sample feature matrices to obtain N enhanced sample feature matrices;

[0217] A second decoding module 1030 is used to decode the N enhanced sample feature matrices to obtain a binary image with the same resolution as the image to be identified; and

[0218] The model optimization module 1040 is used to train a deep learning model using the label information and the sample target extraction results to obtain a target extraction model.

[0219] According to an embodiment of the present disclosure, any multiple modules among the first encoding module 910, the first enhancement module 920, the first decoding module 930 and the result determination module 940, or any multiple modules among the second encoding module 1010, the second enhancement module 1020, the second decoding module 1030 and the model optimization module 1040 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first encoding module 910, the first enhancement module 920, the first decoding module 930 and the result determination module 940, or at least one of the second encoding module 1010, the second enhancement module 1020, the second decoding module 1030 and the model optimization module 1040 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the first encoding module 910, the first enhancement module 920, the first decoding module 930 and the result determination module 940, or at least one of the second encoding module 1010, the second enhancement module 1020, the second decoding module 1030, and the model optimization module 1040 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0220] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0221] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0222] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0223] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A target extraction method, characterized in that: The method includes: Performing N rounds of processing on M first feature matrices corresponding to the image to be recognized to obtain N encoded feature matrices, where the image to be recognized includes multiple target recognition objects, the resolution sizes of the M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer; Performing feature enhancement processing on the N encoded feature matrices to obtain N enhanced feature matrices; Performing decoding processing on the N enhanced feature matrices to obtain a binary image having the same resolution size as the image to be recognized; and Determining a target extraction result based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized, where the target extraction result represents the number of target recognition objects; The performing N rounds of processing on M first feature matrices corresponding to the image to be recognized to obtain N encoded feature matrices includes: In the case of n = 1, respectively performing feature fusion processing on the M first feature matrices in the first round to obtain M fused feature matrices in the first round; Performing context enhancement processing on the M first feature matrices in the first round to obtain M context enhancement matrices in the first round; Fusing the M fused feature matrices and the M context enhancement matrices to obtain M second feature matrices in the second round; In the case of 1 < n ≤ N, respectively performing feature fusion processing on the M second feature matrices in the (n - 1)-th round to obtain M fused feature matrices in the (n - 1)-th round; Performing context enhancement processing on the M second feature matrices in the (n - 1)-th round to obtain M context enhancement matrices in the (n - 1)-th round; Fusing the M fused feature matrices and the context enhancement matrices respectively corresponding to the M fused feature matrices to obtain M second feature matrices in the n-th round; Performing a resampling operation on the M second feature matrices in the n-th round to obtain M resampled feature matrices; and Fusing the M resampled feature matrices to obtain the encoded feature matrix in the (n - 1)-th round.

2. The method according to claim 1, characterized in that The respectively performing feature fusion processing on the M second feature matrices in the (n - 1)-th round to obtain M fused feature matrices in the (n - 1)-th round includes: Performing convolution processing on the i-th second feature matrix in the (n - 1)-th round to obtain the i-th convolutional feature matrix, where i is a positive integer less than or equal to M; Based on M - 1 second feature matrices in the (n - 1)-th round, determining M - 1 to-be-fused feature matrices, where the resolution sizes of the M - 1 to-be-fused feature matrices are the same as that of the i-th convolutional feature matrix; the M - 1 second feature matrices are the other second feature matrices in the (n - 1)-th round except the i-th second feature matrix; and Fusing the i-th convolutional feature matrix and the M - 1 to-be-fused feature matrices to obtain the i-th fused feature matrix.

3. The method according to claim 1, characterized in that The respectively performing context enhancement processing on the M second feature matrices in the (n - 1)-th round to obtain M context enhancement matrices in the (n - 1)-th round includes: Fuse the M second feature matrices of the (n - 1)-th round to obtain the context enhancement matrix to be enhanced in the (n - 1)-th round, where the number of channels of the context enhancement matrix to be enhanced is the sum of the numbers of channels of the M second feature matrices of the (n - 1)-th round respectively; Input the context enhancement matrix to be enhanced in the (n - 1)-th round into the channel attention module of the (n - 1)-th round to obtain the channel enhanced feature matrix of the (n - 1)-th round, where the channel attention module of the (n - 1)-th round is used to adjust the weights of multiple channels of the context enhancement matrix to be enhanced in the (n - 1)-th round; and Resample the channel enhanced feature matrix of the (n - 1)-th round to obtain M context enhancement matrices, where the matrix sizes of the M context enhancement matrices of the (n - 1)-th round correspond one-to-one to the resolution sizes of the M fused feature matrices of the (n - 1)-th round.

4. The method according to claim 1, characterized in that: The feature enhancement processing of the N encoded feature matrices to obtain N enhanced feature matrices includes: Perform downsampling operations on the first j - 1 encoded feature matrices to obtain j - 1 downsampled feature matrices; Fuse the j - 1 downsampled feature matrices with the j - th encoded feature matrix to obtain the j - th initial enhanced feature matrix, where the resolution of the j - th encoded feature matrix is less than the resolutions of the first j - 1 encoded feature matrices, 1 < j ≤ N and j is an integer; and Input the j - th initial enhanced feature matrix into the enhanced channel attention module corresponding to the j - th encoded feature matrix to obtain the j - th enhanced feature matrix, where the enhanced channel attention module corresponding to the j - th encoded feature matrix is used to adjust the weights of multiple channels of the j - th initial enhanced feature matrix.

5. The method according to claim 1, characterized in that The decoding process of the N enhanced feature matrices to obtain a binary image with the same resolution size as the image to be recognized includes: Fuse the N enhanced feature matrices by using the multi-scale feature fusion technology to obtain a multi-scale fusion matrix; Resample the multi-scale fusion matrix to obtain a reconstructed feature matrix; Process the reconstructed feature matrix by using an activation function to obtain a threshold feature matrix, where each pixel value in the threshold feature matrix is between 0 and 1; Perform binarization processing on the threshold feature matrix according to a preset pixel threshold to obtain the binary image.

6. The method according to claim 1, characterized in that The determination of the target extraction result based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized includes: Process the binary image by using an image segmentation algorithm to obtain the number of pixels of the target recognition object in the binary image; Determine the floor area of the target recognition object based on the spatial resolution of the image to be recognized and the number of pixels; Determine the number of the target recognition objects based on the floor area and the average density of the target recognition object.

7. A method for training a target extraction model, characterized in that: The method includes: Perform N rounds of processing on M first sample feature matrices corresponding to the sample image to be recognized to obtain N encoded sample feature matrices, where the sample image to be recognized includes multiple target recognition objects, the sample image to be recognized corresponds to label information, the resolution sizes of the M first sample feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer; Perform feature enhancement processing on the N encoded sample feature matrices to obtain N enhanced sample feature matrices; Perform decoding processing on the N enhanced sample feature matrices to obtain a sample binary image with the same resolution size as the sample image to be recognized; and Use the label information and the sample binary image to train a deep learning model to obtain a target extraction model; The performing N rounds of processing on M first sample feature matrices corresponding to the sample image to be recognized to obtain N encoded sample feature matrices includes: In the case of n = 1, perform feature fusion processing on the M first sample feature matrices in the first round respectively to obtain M fused feature matrices in the first round; Perform context enhancement processing on the M first sample feature matrices in the first round respectively to obtain M context enhancement matrices in the first round; Fuse the M fused feature matrices and the M context enhancement matrices to obtain M second feature matrices in the second round; In the case of 1 < n ≤ N, perform feature fusion processing on the M second feature matrices in the (n - 1)-th round respectively to obtain M fused feature matrices in the (n - 1)-th round; Perform context enhancement processing on the M second feature matrices in the (n - 1)-th round respectively to obtain M context enhancement matrices in the (n - 1)-th round; Fuse the M fused feature matrices and the context enhancement matrices corresponding to the M fused feature matrices respectively to obtain M second feature matrices in the n-th round; Perform a resampling operation on the M second feature matrices in the n-th round to obtain M resampled feature matrices; and Fuse the M resampled feature matrices to obtain the encoded sample feature matrix in the (n - 1)-th round.

8. A target extraction device, characterized in that: The device includes: A first encoding module, configured to perform N rounds of processing on M first feature matrices corresponding to the image to be recognized to obtain N encoded feature matrices, where the image to be recognized includes multiple target recognition objects, the resolution sizes of the M first feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer; A first enhancement module, configured to perform feature enhancement processing on the N encoded feature matrices to obtain N enhanced feature matrices; A first decoding module, configured to perform decoding processing on the N enhanced feature matrices to obtain a binary image with the same resolution size as the image to be recognized; and A result determination module, configured to determine a target extraction result based on the number of pixels of the target recognition object in the binary image and the spatial resolution of the image to be recognized, where the target extraction result represents the number of the target recognition objects; The first encoding module includes: The first fusion sub-module is used to perform feature fusion processing on the M first feature matrices in the first round respectively when n = 1, so as to obtain the M fused feature matrices in the first round; The first enhancement sub-module is used to perform context enhancement processing on the M first feature matrices in the first round respectively, so as to obtain the M context-enhanced matrices in the first round; The first determination sub-module is used to fuse the M fused feature matrices and the M context-enhanced matrices to obtain the M second feature matrices in the second round; The second fusion sub-module is used to perform feature fusion processing on the M second feature matrices in the (n - 1)-th round respectively when 1 < n ≤ N, so as to obtain the M fused feature matrices in the (n - 1)-th round; The second enhancement sub-module is used to perform context enhancement processing on the M second feature matrices in the (n - 1)-th round respectively, so as to obtain the M context-enhanced matrices in the (n - 1)-th round; The second determination sub-module is used to fuse the M fused feature matrices and the context-enhanced matrices corresponding to the M fused feature matrices respectively to obtain the M second feature matrices in the n-th round; The first sampling sub-module is used to perform resampling operations on the M second feature matrices in the n-th round to obtain M resampled feature matrices; The third determination sub-module is used to fuse the M resampled feature matrices to obtain the encoded feature matrix in the (n - 1)-th round.

9. A training device for a target extraction model, characterized in that: The device includes: The second encoding module is used to perform N rounds of processing on the M first sample feature matrices corresponding to the sample image to be recognized to obtain N encoded sample feature matrices, where the sample image to be recognized includes multiple target recognition objects, the sample image to be recognized corresponds to label information, the resolution sizes of the M first sample feature matrices are different from each other, M is an integer greater than 2, and N is a positive integer; The second enhancement module is used to perform feature enhancement processing on the N encoded sample feature matrices to obtain N enhanced sample feature matrices; The second decoding module is used to perform decoding processing on the N enhanced sample feature matrices to obtain a sample binary image with the same resolution size as the sample image to be recognized; and The model optimization module is used to train a deep learning model by using the label information and the sample binary image to obtain a target extraction model; The second encoding module is further used for: When n = 1, performing feature fusion processing on the M first sample feature matrices in the first round respectively to obtain the M fused feature matrices in the first round; Performing context enhancement processing on the M first sample feature matrices in the first round respectively to obtain the M context-enhanced matrices in the first round; Fusing the M fused feature matrices and the M context-enhanced matrices to obtain the M second feature matrices in the second round; When 1 < n ≤ N, performing feature fusion processing on the M second feature matrices in the (n - 1)-th round respectively to obtain the M fused feature matrices in the (n - 1)-th round; Performing context enhancement processing on the M second feature matrices in the (n - 1)-th round respectively to obtain the M context-enhanced matrices in the (n - 1)-th round; The M fused feature matrices and the context enhancement matrices corresponding to the M fused feature matrices are respectively fused to obtain M second feature matrices of the nth round; Resampling the M second feature matrices of the nth round to obtain M resampled feature matrices; and The M resampled feature matrices are merged to obtain the encoding sample feature matrix of the n-1th round.

Images