Vehicle-mounted image classification and effective region segmentation method and device based on semantic segmentation

Abstract

The invention relates to a semantic segmentation-based vehicle-mounted image classification and effective area segmentation method and device, and the method comprises the steps: obtaining a farmland operation vehicle monitoring image, and carrying out the marking of the farmland operation vehicle monitoring image; constructing a neural network for farmland operation vehicle monitoring image classification and effective region segmentation, and encoding the farmland operation vehicle monitoring image; extracting multi-scale feature information of the encoded image; decoding the multi-scale feature information, and outputting semantic information and distance grading information of each pixel point of the decoded image; in the extraction process of sharing multi-scale feature information, semantic information and distance grading information are used for constructing a multi-branch output network; establishing a loss function of the multi-branch output network; training the neural network; and inputting a to-be-processed farmland operation vehicle monitoring image, performing neural network processing, and fusing the output of each branch of the multi-branch output network to obtain a final output result.

Description

Technical Field

[0001] The present invention relates to the field of precision agriculture intelligent decision-making technology, and in particular to the field of vehicle-mounted image classification and effective area segmentation based on semantic segmentation. Background Art

[0002] The on-board monitoring of farmland operation vehicles such as tractors mainly uses the on-board control terminal and the camera to regularly capture the farmland area in front of or behind the tractor to obtain real-time on-board monitoring images. At present, the ownership of ordinary agricultural machinery in my country is close to saturation, and the price of front-mounted intelligent agricultural machinery is high and it is difficult to update. Therefore, the camera is usually installed later to carry out intelligent transformation of agricultural machinery. Because the operating environment of tractors is relatively bumpy, during the on-board monitoring process, the changes in the installation position and angle of the camera will cause the on-board captured image to not be a single farmland scene. When the proportion of other categories of pixels exceeds a certain range, this type of image cannot meet the application requirements of on-board monitoring. Therefore, it is necessary to improve the screening efficiency and image utilization of on-board images.

[0003] Different from conventional image classification methods such as U-Net, RefineNet, SegNet, and DeepLab v3+ segmentation methods, tractor-mounted monitoring images are not divided according to scene categories, but are combined according to the pixel proportions of different scenes, presenting four application states: in the first type of images, the sky accounts for more than 30%, and the angle is too high to meet the monitoring requirements; in the second type of images, the machine accounts for more than 10%, and the angle is too low to meet the monitoring requirements; in the third type of images, the farmland accounts for about 90%, which is a standard monitoring image; in the fourth type of images, the sky accounts for less than 30% and the machine accounts for less than 10%, which is an image to be processed, and the classification and effective area division of the fourth type of images should be improved.

[0004] The pixel-level semantic segmentation method can simultaneously complete the image classification and effective area segmentation tasks. Existing segmentation methods have been successfully applied to many fields, but the research on tractor on-board monitoring is also limited. The natural environment of the tractor on-board monitoring scene is complex and easy to change, and the contours of semantic objects are fuzzy, which affects the accuracy and versatility of the semantic segmentation model. Therefore, on the basis of the codec structure, key methods such as residual units and multi-scale convolution are introduced to parse the category of single pixels in the on-board captured images, and the image classification is realized after counting the proportion of pixels in each category. For images with a small proportion of sky and machinery, the effective area segmentation of farmland can be realized after fusing semantic information and distance information, so that it can meet the application requirements of tractor on-board monitoring images. Summary of the invention

[0005] In order to solve the above problems, the present invention discloses a vehicle-mounted image classification and effective area segmentation method based on semantic segmentation, which is characterized by comprising the following steps:

[0006] Acquire farmland operation vehicle monitoring images, annotate the farmland operation vehicle monitoring images, and obtain a training set;

[0007] Construct a neural network for farmland operation vehicle monitoring image classification and effective area segmentation, where:

[0008] Encoding the farmland operation vehicle monitoring image to obtain an encoded image;

[0009] Extracting multi-scale feature information of the encoded image;

[0010] Decoding the multi-scale feature information, and outputting semantic information and distance classification information of each pixel of the decoded image;

[0011] Sharing the extraction process of the multi-scale feature information, using the semantic information and distance classification information to construct a multi-branch output network;

[0012] Establishing a loss function of the multi-branch output network;

[0013] Training the neural network using the training set;

[0014] The farmland operation vehicle monitoring image to be processed is input, processed by the neural network, and the outputs of each branch of the multi-branch output network are integrated to obtain the final output result.

[0015] In one embodiment of the above method of the present invention, the step of acquiring the monitoring image of the farmland operation vehicle further comprises:

[0016] A camera is installed at the front of the roof of the cab of a farmland operation vehicle, along the straight forward direction of the vehicle and parallel to the horizontal plane, at an initial installation angle inclined toward the ground, to obtain images of farmland within a range in front of the vehicle.

[0017] In one embodiment of the above method of the present invention, the initial installation angle is 0 degrees, the tilt angle is 60 degrees, and the range is 15 meters.

[0018] In an embodiment of the above method of the present invention, the camera is any one of a monocular camera, a binocular camera, a stereo 3D camera, and an infrared camera.

[0019] In one embodiment of the above method of the present invention, the step of encoding the farmland operation vehicle monitoring image is performed based on a lightweight residual network and a hybrid void convolution.

[0020] In one embodiment of the above method of the present invention, the step of encoding the farmland operation vehicle monitoring image based on the lightweight residual network and the hybrid void convolution further includes:

[0021] Convolve the input image according to the number of channels to obtain a feature map;

[0022] Downsampling the feature map;

[0023] Dividing the feature graph into multiple branches according to a feature number to be extracted;

[0024] According to the separable characteristics of fast Gaussian filtering, the two-dimensional convolution kernel is split into two one-dimensional convolution kernels in different directions;

[0025] Performing mixed hole convolution on the divided branch feature map to increase the receptive field and enhance local information;

[0026] fusing the multi-branch feature maps;

[0027] Performing a jump connection between the input image and the fused multi-branch feature map to restore the original number of channels;

[0028] The image after the skip connection is subjected to channel shuffle.

[0029] In one embodiment of the above method of the present invention, the dilation rate of the extracted mixed dilated convolution is 2.

[0030] In one embodiment of the above method of the present invention, wherein the monitoring image of the farmland operation vehicle

[0031] Enter

[0032] Row encoding can be performed multiple times, and the computational complexity of the encoding is:

[0033]

[0034] Among them, is the number of input channels of layer l, C l is the number of output channels of layer l; K lx and K ly is the size of the convolution kernel, K lx and K ly The size of the feature map input to layer l.

[0035] In one embodiment of the above method of the present invention, the step of extracting multi-scale feature information of the encoded image further comprises:

[0036] Dividing the encoded image into a multi-scale convolution branch and a common convolution branch;

[0037] Performing a convolution operation on the common convolution branch using a 3×3 convolution kernel;

[0038] Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 3×3 convolution operation of the common convolution branch to extract the single-scale first target quantity feature;

[0039] Performing a first downsampling on the multi-scale convolution branch;

[0040] Performing a convolution operation on the result after the first downsampling using a 7×7 convolution kernel to obtain a 7×7 convolution operation result;

[0041] Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 7×7 convolution operation to extract the second target quantity feature at the first scale;

[0042] Performing a second downsampling on the 7×7 convolution operation result;

[0043] The result after the second downsampling is then convolved using a 5×5 convolution kernel to obtain a 5×5 convolution operation result;

[0044] Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 5×5 convolution operation to extract the second target quantity feature at the second scale;

[0045] Performing a third downsampling on the 5×5 convolution operation result;

[0046] The result of the third downsampling is then convolved using a 3×3 convolution kernel to obtain a 3×3 convolution operation result;

[0047] A convolution dimension reduction operation with a convolution kernel of 3×3 is performed on the result of the 3×3 convolution operation to extract the second target quantity feature at the third scale.

[0048] In one embodiment of the above method of the present invention, the downsampling step is to perform a pooling operation using a 2×2 matrix.

[0049] In an embodiment of the above method of the present invention, the pooling operation includes average pooling and maximum pooling.

[0050] In an embodiment of the above method of the present invention, the step of decoding the multi-scale feature information further comprises:

[0051] Using a 1×1 convolution kernel to perform feature fusion on the single-scale first target quantity feature to obtain a decoding result of a common convolution branch;

[0052] Performing a first upsampling of the second target quantity feature at the third scale;

[0053] Performing a first feature fusion on the result of the first upsampling and the second target quantity feature at the second scale using a 1×1 convolution kernel;

[0054] Performing a second upsampling on the result of the first feature fusion;

[0055] Performing a second feature fusion on the result of the second upsampling and the second target quantity feature at the first scale using a 1×1 convolution kernel;

[0056] The result of the second feature fusion is upsampled for the third time to obtain a decoding result of the multi-scale convolution branch.

[0057] In one embodiment of the above method of the present invention, the step of constructing a multi-branch output network further comprises:

[0058] Perform dot product operations on the decoding results of the multi-scale convolution branch and the ordinary convolution branch to merge;

[0059] Performing an interpolation operation on the combined result of the dot product operation to obtain a semantic part including the second target quantity and a distance part including one channel;

[0060] Performing global decoding of a second target quantity feature on the encoded image;

[0061] Performing a sum operation on the image after the global decoding of the second target quantity feature and the semantic part of the second target quantity channel to obtain an output of the semantic branch;

[0062] Performing global decoding of distance features on the encoded image;

[0063] The distance feature global decoded image and the distance part of the one channel are summed to obtain the output of the distance classification branch.

[0064] In an embodiment of the above method of the present invention, the first target number is 4, and the second target number is 3.

[0065] In one embodiment of the above method of the present invention, the step of global decoding further comprises:

[0066] Performing a global average pooling operation on the encoded image;

[0067] Performing a 1×1 convolution operation on the result of the global average pooling operation;

[0068] The result after the 1×1 convolution operation is upsampled.

[0069] In one embodiment of the above method of the present invention, the upsampling step is to perform an anti-pooling operation using a bilinear interpolation method.

[0070] In one embodiment of the above method of the present invention, the step of establishing the loss function of the multi-branch output network further comprises:

[0071] Using a classification cross entropy loss function for the semantic branch;

[0072] The reverse Huber loss function is used for the distance classification branch.

[0073] The present invention also discloses a vehicle-mounted image classification and effective area segmentation device based on semantic segmentation, which is used to implement any of the above methods, including:

[0074] Image monitoring module, used to obtain monitoring images of farmland operation vehicles;

[0075] An encoding module encodes the farmland operation vehicle monitoring image to obtain an encoded image;

[0076] A multi-scale feature extraction module, used to extract multi-scale feature information of the encoded image;

[0077] A decoding module, used for decoding the multi-scale feature information and the encoded image;

[0078] A multi-branch output module, used to use the distance classification information as an auxiliary identifier to construct a multi-branch output network;

[0079] The model optimization module is used to establish the loss function of the multi-branch output network and optimize the neural network parameters.

[0080] The present invention also discloses a farmland operation vehicle, which is characterized in that it includes a vehicle body installed on a frame, a vehicle-mounted control system and a driving device, and the vehicle-mounted control system further includes: the above-mentioned vehicle-mounted image classification and effective area segmentation device based on semantic segmentation.

[0081] In one embodiment of the above method of the present invention, the computer program is used to execute the steps of any one of the above methods.

[0082] Based on the above, through the method and device of vehicle-mounted image classification and effective area segmentation based on semantic segmentation disclosed by the present invention, semantic information and distance information can be used to remove blurred farmland in the long distance, not only the category of pixels can be judged, but also the distance classification result can be given, thereby realizing the effective area segmentation of farmland, improving the application efficiency of vehicle-mounted monitoring means of farmland operation vehicles, and allowing a large number of non-standard images to be used as the calculation basis for quantitative analysis of vehicle-mounted monitoring, providing effective technical support for realizing intelligent decision-making in precision agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1The figure is a flow chart of a method for classifying images and segmenting effective areas of farmland operation vehicle monitoring images based on semantic segmentation in one embodiment of the present invention.

[0084] Figure 2 The present invention is a schematic block diagram of the architecture of a method for classifying farmland operation vehicle monitoring images and segmenting effective areas based on semantic segmentation in one embodiment of the present invention.

[0085] Figure 3 The present invention is a flowchart of the steps of encoding a monitoring image of a farmland operation vehicle in one embodiment of the present invention.

[0086] Figure 4 The figure is a flowchart of the steps of multi-scale feature extraction and multi-scale feature decoding in one embodiment of the present invention.

[0087] Figure 5 Schematic diagram of input image and output result of a method for classifying farmland operation vehicle monitoring images and segmenting effective areas based on semantic segmentation in one embodiment of the present invention.

[0088] Figure 6 It is a structural diagram of a device for classifying images and segmenting effective areas of farmland operation vehicle monitoring based on semantic segmentation in one embodiment of the present invention.

[0089] Figure 7 The present invention is a structural block diagram of a farmland operation vehicle in one embodiment of the present invention.

[0090] Wherein, the reference numerals are:

[0091] 1: Farmland operation vehicles

[0092] 2: Farmland

[0093] 10: Neural network for image classification and effective area segmentation of farmland operation vehicle monitoring

[0094] 100: Encoding

[0095] 1001: Convolution

[0096] 1002: Downsampling

[0097] 1003: Residual Network

[0098] 101: Multi-scale feature extraction

[0099] 102: Multi-scale feature decoding

[0100] 103: Global decoding of the second target quantity feature

[0101] 104: Global decoding of distance features

[0102] 105: Insertion and subtraction

[0103] 106: Addition

[0104] 11: Farmland operation vehicle monitoring image

[0105] 12: Image classification results

[0106] 13: Distance classification results

[0107] 20: Vehicle monitoring image classification and effective area segmentation device based on semantic segmentation

[0108] 201: Image detection module

[0109] 202: Encoding module

[0110] 203: Multi-scale feature extraction module

[0111] 204: Decoding module

[0112] 205: Multi-branch output module

[0113] 206: Model Optimization Module

[0114] 30: Vehicle control system

[0115] 40: Frame

[0116] 50: Car body

[0117] 60: Drive device DETAILED DESCRIPTION

[0118] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments to further understand the purpose, solution and beneficial technical effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0119] It should be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0120] Certain words are used in the specification and the appended claims to refer to specific components or parts. It should be understood by those skilled in the art that technical users or manufacturers may refer to the same component or part by different nouns or terms. This specification and the appended claims do not distinguish components or parts by differences in names, but by differences in functions.

[0121] In the present invention, the directions or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like are based on the directions or positional relationships shown in the drawings. These terms are mainly used to better describe the present invention and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific direction, or to be constructed and operated in a specific direction.

[0122] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those skilled in the art, the specific meanings of these terms in the present invention can be understood according to specific circumstances.

[0123] In addition, the terms "installed", "set", "provided with", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be an internal connection between two devices, elements, or components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0124] Please refer to Figure 1 and Figure 2 , Figure 1 The figure is a flow chart of a method for classifying images and segmenting effective areas of farmland operation vehicle monitoring images based on semantic segmentation in one embodiment of the present invention. Figure 2 The present invention is a schematic block diagram of the architecture of a method for classifying farmland operation vehicle monitoring images and segmenting effective areas based on semantic segmentation in one embodiment of the present invention.

[0125] The present invention discloses a vehicle-mounted image classification and effective area segmentation method based on semantic segmentation, comprising the following steps:

[0126] S100: Acquire monitoring images of farmland operation vehicles, and annotate the monitoring images of farmland operation vehicles to obtain a training set;

[0127] S200: constructing a neural network 10 for classifying farmland operation vehicle monitoring images and segmenting effective areas, wherein the farmland operation vehicle monitoring images are encoded to obtain encoded images;

[0128] S300: extracting multi-scale feature information of the encoded image;

[0129] S400: Decoding the multi-scale feature information, and outputting semantic information and distance classification information of each pixel of the decoded image;

[0130] S500: Sharing the extraction process of the multi-scale feature information, using the semantic information and distance classification information to construct a multi-branch output network;

[0131] S600: Establishing a loss function of the multi-branch output network;

[0132] S700: Training the neural network using the training set;

[0133] S800: Input the farmland operation vehicle monitoring image to be processed, process it through the neural network, fuse the outputs of each branch of the multi-branch output network, and obtain the final output result.

[0134] In the above steps, the images of farmland operation vehicle monitoring can be annotated using LABELME software and saved as JSON files. When applied, they are converted into single-channel images for easy model reading.

[0135] The above-mentioned step of obtaining the monitoring image of the farmland operation vehicle further includes:

[0136] A camera is installed at the front of the roof of the cab of the farmland operation vehicle, along the straight forward direction of the vehicle and parallel to the horizontal plane, at an initial installation angle and tilted toward the ground by an angle, to obtain images of farmland within a range in front of the vehicle. The initial installation angle may be 0 degrees, the tilt angle may be 60 degrees, and the range may be 15 meters, but the present invention is not limited thereto.

[0137] The camera may be any one of a monocular camera, a binocular camera, a stereo 3D camera, and an infrared camera.

[0138] The steps of creating the neural network 10 for the classification of farmland operation vehicle monitoring images and effective area segmentation include S200 to S600, wherein, Figure 2 As shown, including:

[0139] The monitoring image of the farmland operation vehicle is encoded 100; the encoding process includes convolution 1001 operation, down sampling 1002 and residual network 1003 processing;

[0140] Perform multi-scale feature extraction on the encoded image 101;

[0141] Performing multi-scale feature decoding 102 on the result of multi-scale feature extraction 101 to obtain second target quantity semantic information and distance classification information;

[0142] The result of multi-scale feature decoding 102 is interpolated 105 to separate the second target quantity semantic information and the distance classification information;

[0143] Performing a summing operation 106 on the result of the global decoding 103 of the second target quantity feature and the second target quantity semantic information, performing a summing operation 106 on the result of the global decoding 104 of the distance feature and the second target quantity semantic information, and constructing a multi-branch output network;

[0144] Establish the loss function of the multi-branch output network ( Figure 2 not shown).

[0145] Please refer to Figure 3 , Figure 3 The flowchart of the steps of encoding the detection image of the farmland operation vehicle in one embodiment of the present invention is as follows. The above-mentioned step of encoding the monitoring image of the farmland operation vehicle is performed based on the lightweight residual network and the mixed void convolution.

[0146] The above-mentioned step of encoding the farmland operation vehicle monitoring image based on the lightweight residual network and the hybrid void convolution further includes:

[0147] S201: Convolve the input image according to a number of channels to obtain a feature map;

[0148] S202: Downsample the feature map, and the size of the downsampled image is M lx ×M ly ;

[0149] S203: Dividing the feature graph into multiple branches according to a feature number to be extracted;

[0150] S204: splitting the two-dimensional convolution kernel into two one-dimensional convolution kernels in different directions according to the separable characteristic of the fast Gaussian filter;

[0151] S205: performing mixed dilated convolution on the divided branch feature map to increase the receptive field and enhance local information;

[0152] S206: fusing the multi-branch feature graphs;

[0153] S207: Performing a jump connection between the input image and the fused multi-branch feature map to restore the original number of channels;

[0154] S208: Shuffling the channels of the jump-connected images.

[0155] The output image size is M lx ×M ly .

[0156] The dilation rate of the above hybrid dilated convolution can be 2.

[0157] The above encoding of the farmland operation vehicle monitoring image can be performed multiple times, and the encoding calculation

[0158] The amount is:

[0159]

[0160] Among them, is the number of input channels of layer l, C l is the number of output channels of layer l; K lx and K ly is the size of the convolution kernel, M lx and M ly The size of the feature map input to layer l.

[0161] like Figure 1 As shown, the size of the farmland operation vehicle monitoring image is, for example, 320×240. The farmland operation vehicle monitoring image can be convolved using a convolution kernel of 3×3 and 32 channels for simultaneous convolution. The image size and the number of channels after convolution are 320×240×32. The second convolution operation is performed using a convolution kernel of 3×3 and 64 channels. The image size and the number of channels after convolution are 160×120×64. The third convolution operation is performed using a convolution kernel of 3×3 and 128 channels. The image size and the number of channels after convolution are 80×60×128. A total of three encoding processes are performed to complete the encoding of the input image, wherein the size of the monitoring image, the size of the convolution kernel, the number of channels and the number of encodings are all schematic diagrams for better illustrating the present invention, and the present invention is not limited thereto.

[0162] like Figure 4 As shown, Figure 4 The figure is a flowchart of the steps of multi-scale feature extraction and multi-scale feature decoding in one embodiment of the present invention.

[0163] After encoding the farmland operation vehicle monitoring image, it is necessary to extract multi-scale feature information of the encoded image, and the step of extracting the multi-scale feature information further includes:

[0164] The input feature map size is, for example, 80×60×128; a 3×3 convolution kernel is used for convolution as Conv-3, a 5×5 convolution kernel is used for convolution as Conv-5, and a 7×7 convolution kernel is used for convolution as Conv-7.

[0165] The encoded image, i.e., the input feature map, is divided into a multi-scale convolution branch and a common convolution branch;

[0166] The common convolution branch is convolved using a 3×3 convolution kernel to obtain a feature map with a size of 80×60×128;

[0167] A convolution dimension reduction operation with a convolution kernel of 3×3 is performed on the convolution operation result of the ordinary convolution branch to extract the single-scale first target quantity feature, and the feature map size is 80×60×(3+1).

[0168] The multi-scale convolution branch is downsampled for the first time, and the feature map size is 40×30×128 at this time;

[0169] The result after the first downsampling is convolved with a 7×7 convolution kernel to obtain a 7×7 convolution operation result with a feature map size of 40×30×128;

[0170] Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 7×7 convolution operation to extract the second target quantity feature at the first scale, and the feature map size is 40×30×3;

[0171] The 7×7 convolution operation result is downsampled for the second time, and the feature map size is now 20×15×128;

[0172] The result after the second downsampling is convolved again using a 5×5 convolution kernel to obtain a 5×5 convolution operation result, and the feature map size is 20×15×128;

[0173] Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 5×5 convolution operation to extract the second target quantity feature at the second scale, with a feature map size of 20×15×3;

[0174] The 5×5 convolution operation result is downsampled for the third time, and the feature map size is now 10×8×128;

[0175] The result of the third downsampling is convolved with a 3×3 convolution kernel to obtain a 3×3 convolution operation result with a feature map size of 10×8×128.

[0176] A convolution dimension reduction operation with a convolution kernel of 3×3 is performed on the result of the 3×3 convolution operation to extract the second target quantity features at the third scale, and the feature map size is 10×8×3.

[0177] Downsampling is done by pooling using a 2×2 matrix.

[0178] The above pooling operations include average pooling and maximum pooling.

[0179] After extracting the multi-scale feature information, the multi-scale feature information is decoded, and the step of decoding the multi-scale feature information further includes:

[0180] A 1×1 convolution kernel is used to perform feature fusion on the common convolution branch to obtain a decoding result of the common convolution branch, and the feature map size is 80×60×(3+1);

[0181] Performing a first upsampling on the second target quantity feature at the third scale, where the feature map size is 20×15×3;

[0182] Performing a first feature fusion on the result of the first upsampling and the second target quantity feature at the second scale using a 1×1 convolution kernel;

[0183] The result of the first feature fusion is upsampled for the second time, and the feature map size is 40×30×3 at this time;

[0184] A second feature fusion is performed on the result of the second upsampling and the second target quantity feature at the first scale using a 1×1 convolution kernel.

[0185] The result of the second feature fusion is upsampled for the third time to obtain the decoding result of the multi-scale convolution branch. At this time, the feature map size is 80×60×3.

[0186] The above-mentioned input feature map size, convolution kernel size, number of channels, number of extracted features, number of downsampling and upsampling are all for better explanation of the present invention, and the present invention is not limited thereto.

[0187] After decoding the multi-scale features, a multi-branch output network is constructed, and the construction steps further include:

[0188] Perform dot product operations on the decoding results of the multi-scale convolution branch and the ordinary convolution branch to merge;

[0189] Performing an interpolation operation on the combined result of the dot product operation to obtain a semantic part including the second target number of channels and a distance part including one channel;

[0190] Performing global decoding of a second target quantity feature on the encoded image;

[0191] Performing a sum operation on the image after the global decoding of the second target quantity feature and the semantic part of the second target quantity channel to obtain an output of the semantic branch;

[0192] Performing global decoding of distance features on the encoded image;

[0193] The distance feature global decoded image and the distance part of the one channel are summed to obtain the output of the distance classification branch.

[0194] It should be noted that the number of first targets in the above-mentioned multi-scale feature extraction, multi-scale feature decoding and construction of a multi-branch output network can be 4, and the number of second targets can be 3, but the present invention is not limited thereto.

[0195] The above global decoding step further includes:

[0196] Performing a global average pooling operation on the encoded image;

[0197] Performing a 1×1 convolution operation on the result of the global average pooling operation;

[0198] The result after the 1×1 convolution operation is upsampled.

[0199] In addition, the above upsampling step uses a bilinear interpolation method to perform an unpooling operation.

[0200] Next, a loss function of the multi-branch output network is established, and the step of establishing the loss function of the multi-branch output network further includes:

[0201] Using a classification cross entropy loss function for the semantic branch;

[0202] The reverse Huber loss function is used for the distance classification branch.

[0203] Please refer to Figure 5 , Figure 5 Schematic diagram of input image and output result of a method for classifying farmland operation vehicle monitoring images and segmenting effective areas based on semantic segmentation in one embodiment of the present invention.

[0204] The actual test set, namely the farmland operation vehicle monitoring image 11, is processed by the neural network 10 for farmland operation vehicle monitoring image classification and effective area segmentation to obtain an image classification result 12 and a distance classification result 13.

[0205] like Figure 6 As shown, the present invention also discloses a vehicle monitoring image classification and effective area segmentation device 20 based on semantic segmentation, which is used to implement any of the above methods, including:

[0206] Image monitoring module 201, used to obtain monitoring images of farmland operation vehicles;

[0207] The encoding module 202 encodes the farmland operation vehicle monitoring image to obtain an encoded image;

[0208] A multi-scale feature extraction module 203, used to extract multi-scale feature information of the encoded image;

[0209] A decoding module 204, configured to decode the multi-scale feature information and the encoded image;

[0210] A multi-branch output module 205, configured to use the distance classification information as an auxiliary identifier to construct a multi-branch output network;

[0211] The model optimization module 206 is used to establish the loss function of the multi-branch output network and optimize the neural network parameters.

[0212] like Figure 7 As shown, the present invention also discloses a farmland operation vehicle 1, comprising a vehicle body 50 mounted on a frame 40, a vehicle-mounted control system 30 and a driving device 60, wherein the vehicle-mounted control system 30 further comprises: the above-mentioned vehicle-mounted vehicle monitoring image classification and effective area segmentation device 20 based on semantic segmentation. The farmland operation vehicle 1 performs semantic classification and effective area division on the farmland 2 through the vehicle-mounted monitoring image classification and effective area segmentation device 20 based on semantic segmentation.

[0213] The present invention also discloses a storage medium for storing a computer program, wherein the computer program is used to execute the steps of any one of the above methods.

[0214] In summary, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, technicians familiar with the field can evolve various corresponding changes and deformations based on the present invention, but these corresponding changes and deformations should all fall within the scope of protection of the patent applied for the present invention.

Claims

1. A vehicle image classification and effective area segmentation method based on semantic segmentation, characterized in that: The following steps are involved: Acquire farmland operation vehicle monitoring images, annotate the farmland operation vehicle monitoring images, and obtain a training set; Construct a neural network for farmland operation vehicle monitoring image classification and effective area segmentation, where: Encoding the farmland operation vehicle monitoring image to obtain an encoded image; Extracting multi-scale feature information of the encoded image; Decoding the multi-scale feature information, and outputting semantic information and distance classification information of each pixel of the decoded image; Sharing the extraction process of the multi-scale feature information, using the semantic information and distance classification information to construct a multi-branch output network; Establishing a loss function of the multi-branch output network; Training the neural network using the training set; The farmland operation vehicle monitoring image to be processed is input, processed by the neural network, and the outputs of each branch of the multi-branch output network are integrated to obtain the final output result.

2. The method according to claim 1, characterized in that The step of obtaining the monitoring image of the farmland operation vehicle further includes: A camera is installed at the front of the roof of the cab of a farmland operation vehicle, along the straight forward direction of the vehicle and parallel to the horizontal plane, at an initial installation angle inclined toward the ground, to obtain images of farmland within a range in front of the vehicle.

3. The method according to claim 2, characterized in that The initial installation angle is 0 degrees, the tilt angle is 60 degrees, and the range is 15 meters.

4. The method according to claim 3, characterized in that The camera is any one of a monocular camera, a binocular camera, a stereo 3D camera, and an infrared camera.

5. The method according to claim 1, characterized in that The step of encoding the farmland operation vehicle monitoring image is based on a lightweight residual network and a mixed void convolution.

6. The method according to claim 5, characterized in that The step of encoding the farmland operation vehicle monitoring image based on the lightweight residual network and the hybrid void convolution further includes: Convolve the input image according to the number of channels to obtain a feature map; Downsampling the feature map; Dividing the feature graph into multiple branches according to a feature number to be extracted; According to the separable characteristics of fast Gaussian filtering, the two-dimensional convolution kernel is split into two one-dimensional convolution kernels in different directions; Performing mixed hole convolution on the divided branch feature map to increase the receptive field and enhance local information; fusing the multi-branch feature maps; Performing a jump connection between the input image and the fused multi-branch feature map to restore the original number of channels; The image after the skip connection is subjected to channel shuffle.

7. The method according to claim 6, characterized in that The dilation rate of the mixed dilated convolution is extracted to be 2.

8. The method according to claim 6 or 7, characterized in that The encoding of the farmland operation vehicle monitoring image can be performed multiple times, and the calculation amount of the encoding is: Among them, is the number of input channels of layer l, C l is the number of output channels of layer l; K lx and K ly is the size of the convolution kernel, M lx and M ly The size of the feature map input to layer l.

9. The method according to claim 1, characterized in that The step of extracting multi-scale feature information of the encoded image further comprises: Dividing the encoded image into a multi-scale convolution branch and a common convolution branch; Performing a convolution operation on the common convolution branch using a 3×3 convolution kernel; Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 3×3 convolution operation of the common convolution branch to extract the single-scale first target quantity feature; Performing a first downsampling on the multi-scale convolution branch; Performing a convolution operation on the result after the first downsampling using a 7×7 convolution kernel to obtain a 7×7 convolution operation result; Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 7×7 convolution operation to extract the second target quantity feature at the first scale; Performing a second downsampling on the 7×7 convolution operation result; The result after the second downsampling is then convolved using a 5×5 convolution kernel to obtain a 5×5 convolution operation result; Performing a convolution dimension reduction operation with a convolution kernel of 3×3 on the result of the 5×5 convolution operation to extract the second target quantity feature at the second scale; Performing a third downsampling on the 5×5 convolution operation result; The result of the third downsampling is then convolved using a 3×3 convolution kernel to obtain a 3×3 convolution operation result; A convolution dimension reduction operation with a convolution kernel of 3×3 is performed on the result of the 3×3 convolution operation to extract the second target quantity feature at the third scale.

10. The method according to claim 6 or 9, characterized in that The downsampling step is to perform a pooling operation using a 2×2 matrix.

11. The method according to claim 10, characterized in that The pooling operation includes average pooling and maximum pooling.

12. The method according to claim 9, characterized in that The step of decoding the multi-scale feature information further comprises: Using a 1×1 convolution kernel to perform feature fusion on the single-scale first target quantity feature to obtain a decoding result of a common convolution branch; Performing a first upsampling of the second target quantity feature at the third scale; Performing a first feature fusion on the result of the first upsampling and the second target quantity feature at the second scale using a 1×1 convolution kernel; Performing a second upsampling on the result of the first feature fusion; Performing a second feature fusion on the result of the second upsampling and the second target quantity feature at the first scale using a 1×1 convolution kernel; The result of the second feature fusion is upsampled for the third time to obtain a decoding result of the multi-scale convolution branch.

13. The method according to claim 12, characterized in that The step of constructing a multi-branch output network further includes: Perform dot product operations on the decoding results of the multi-scale convolution branch and the ordinary convolution branch to merge; Performing an interpolation operation on the combined result of the dot product operation to obtain a semantic part including the second target number of channels and a distance part including one channel; Performing global decoding of a second target quantity feature on the encoded image; Performing a sum operation on the image after the global decoding of the second target quantity feature and the semantic part of the second target quantity channel to obtain an output of the semantic branch; Performing global decoding of distance features on the encoded image; The distance feature global decoded image and the distance part of the one channel are summed to obtain the output of the distance classification branch.

14. The method according to claim 9, 12 or 13, characterized in that: The first target number is 4, and the second target number is 3.

15. The method according to claim 13, characterized in that The step of global decoding further comprises: Performing a global average pooling operation on the encoded image; Performing a 1×1 convolution operation on the result of the global average pooling operation; The result after the 1×1 convolution operation is upsampled.

16. The method according to claim 12 or 15, characterized in that The upsampling step is to perform an anti-pooling operation using a bilinear interpolation method.

17. The method according to claim 13, characterized in that The step of establishing the loss function of the multi-branch output network further comprises: Using a classification cross entropy loss function for the semantic branch; The reverse Huber loss function is used for the distance classification branch.

18. A vehicle-mounted image classification and effective area segmentation device based on semantic segmentation, used to implement the method according to any one of claims 1 to 17, characterized in that: include: Image monitoring module, used to obtain monitoring images of farmland operation vehicles; An encoding module encodes the farmland operation vehicle monitoring image to obtain an encoded image; A multi-scale feature extraction module, used to extract multi-scale feature information of the encoded image; A decoding module, used for decoding the multi-scale feature information and the encoded image; A multi-branch output module, used to use the distance classification information as an auxiliary identifier to construct a multi-branch output network; The model optimization module is used to establish the loss function of the multi-branch output network and optimize the neural network parameters.

19. A farmland operation vehicle, characterized in that: It comprises a vehicle body mounted on a vehicle frame, a vehicle-mounted control system and a driving device, wherein the vehicle-mounted control system further comprises: a device as claimed in claim 18.

20. A storage medium for storing a computer control program, characterized in that: The computer control program is used to execute the steps of the method according to any one of claims 1 to 17.

Images