A vehicle-mounted image farmland target detection method and device

Abstract

The application provides a vehicle-mounted image farmland target detection method, comprising: constructing a convolutional neural network target detection model of coding and decoding structure, which comprises: extracting multi-level original image features; performing color perception on the multi-level original image features; performing semantic fusion on the multi-level original image features and the multi-level original image features after color perception; performing cavity convolution on the features after semantic fusion; performing spatial enhancement processing on the features after cavity convolution, which is used for preventing large target segmentation boundary overflow and simultaneously having obvious segmentation features for small targets; performing channel fusion on the features after spatial enhancement processing and the features after semantic fusion; performing up-sampling on the features after channel fusion to restore original image size; predicting image segmentation results pixel by pixel for the feature map with restored original image size to obtain target detection results; training the target detection model; and inputting a to-be-detected image into the trained target detection model for target detection.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation, and in particular to image semantic segmentation of farmland in vehicle-mounted images. Background Technology

[0002] Agricultural machinery needs to detect objects such as straw during field operations. Existing technologies, through data cleaning and image processing algorithms, eliminate the interference of hardware facilities on the image quality of straw monitoring from both the imaging position and imaging field of view aspects, providing accurate field monitoring range and effective imaging field of view for straw target detection. However, some uncontrollable human interference factors, such as changes in outdoor light, object occlusion, and the degree of straw damage, can easily cause misclassification of straw and farmland, negatively impacting the accuracy of straw target detection. Although semantic segmentation can distinguish the category of each pixel, straw targets in complex environments are small, semantically ambiguous, and lack obvious edge and depth information, rendering many improved methods based on multimodal fusion of edge and depth features ineffective. Therefore, improving the accuracy of straw target detection in complex environments is one of the important aspects of straw monitoring technology research. Summary of the Invention

[0003] To address the challenges posed by the complex field environment of no-till farming, where straw targets are small and semantically ambiguous, and where external and internal interference factors easily lead to misjudgment of straw pixels in images, this invention proposes a vehicle-mounted image-based farmland target detection method, comprising the following steps:

[0004] Constructing a convolutional neural network object detection model with an encoding / decoding structure, including:

[0005] Extract features from the multi-level original image;

[0006] Color perception is performed on the multi-level features of the original image;

[0007] Semantic fusion is performed between the multi-level original image features and the color-perceived multi-level original image features;

[0008] Perform dilated convolution on the semantically fused features;

[0009] Spatial augmentation is performed on the features after dilated convolution to prevent the segmentation boundary of large targets from overflowing, while small targets have obvious segmentation features.

[0010] Channel fusion is performed between the spatially enhanced features and the semantically fused features.

[0011] Upsample the features after channel fusion to restore the original image size;

[0012] The image segmentation result is predicted pixel by pixel from the feature map that restores the original image size, and the target detection result is obtained.

[0013] The target detection model is trained;

[0014] The image to be detected is input into the trained object detection model for object detection.

[0015] In one embodiment of the method of the present invention, the step of extracting multi-level original image features further includes:

[0016] A convolution operation is performed on the input image of each level of original image features, where the input image of the next level of original image features is the original image features of the previous level, and the input image of the first level of original image features is the original input image.

[0017] In one embodiment of the method of the present invention, the color perception of the multi-level original image features is achieved by feature fusion between the multi-level original image features and the color enhancement features corresponding to each of the multi-level original image features.

[0018] In one embodiment of the method of the present invention, the step of extracting the color enhancement features corresponding to each of the multi-level original image features further includes:

[0019] A convolution operation is performed on the input image of each level of color enhancement feature. The input image of the next level of color enhancement feature is the output feature map of the previous level of color enhancement feature after color perception. The input image of the first level of color enhancement feature is the image after white balance processing of the original input image.

[0020] In one embodiment of the method of the present invention, the color perception step achieved by feature fusion between the original image features and the color enhancement features corresponding to the original image features further includes:

[0021] The original image features and the color enhancement features corresponding to the original image features are subjected to the first set of nonlinear mappings to obtain the nonlinear mapped original image features and the nonlinear mapped color enhancement features.

[0022] The original image features are multiplied by the nonlinear mapping color enhancement features, and then the multiplied product is superimposed on the original image features using the formula. It means that among them This represents the features of the original image after overlay. This represents the original feature map fused with the i-th feature. This represents the nonlinear mapping color enhancement feature fused from the i-th feature;

[0023] The color enhancement feature is multiplied by the original nonlinear mapping feature, and then superimposed on the color enhancement feature using the formula. It means that among them This indicates the color enhancement feature after superposition. This represents the color-enhanced feature map fused from the i-th feature. This represents the nonlinear original image feature fused from the i-th feature;

[0024] Channel fusion is performed on the superimposed original image features and the superimposed color enhancement features, and a second set of nonlinear mapping is applied to the channel fused features.

[0025] In one embodiment of the method of the present invention, the step of performing a first set of nonlinear mappings on the original image features and the color enhancement features corresponding to the original image features further includes:

[0026] Apply a nonlinear mapping to the features of the original image using the formula. express,

[0027] Apply a nonlinear mapping to the color enhancement features using the formula

[0028] In this diagram, f() represents the first convolution operation, BN represents the batch normalization operation, and the nonlinear mapping uses the Sigmoid activation function.

[0029] Before performing the first convolution operation and Pixels are filled on the outer side.

[0030] In one embodiment of the method of the present invention, the step of performing a second set of nonlinear mapping on the channel-fused features further includes:

[0031] Where f′() represents the second convolution operation, concat() represents the channel fusion operation, BN represents the batch normalization operation, and the nonlinear mapping uses the ReLU activation function. This represents the multi-level original image features after color perception.

[0032] In one embodiment of the method of the present invention, the spatial enhancement processing of the features after dilated convolution is achieved by enhancing the spatial positional relationship of targets at different scales through bilinear interpolation upsampling.

[0033] In one embodiment of the method of the present invention, the spatial enhancement processing step of strengthening the spatial positional relationship of targets at different scales through bilinear interpolation upsampling further includes:

[0034] The features after the dilated convolution are subjected to bilinear interpolation upsampling, and then a third set of nonlinear mappings is applied to the features after the bilinear interpolation upsampling to obtain a spatial region mask, which is then expressed using the formula...

[0035] M j=Sigmoid(Up(Down(f″(F)) j )))) represents, where f″() represents the third convolution operation, F j This represents the entire feature map after dilated convolution. Down() represents the downsampling operation, Up() represents the bilinear interpolation upsampling operation, and M... j Indicates a spatial region mask;

[0036] The feature map after dilated convolution is multiplied by the spatial region mask, and the multiplied feature map is then superimposed on the feature map after dilated convolution, using the formula... It means that among them This represents the features after the spatial enhancement process. The feature map represents the multi-scale features of each channel of the dilated convolution output.

[0037] In one embodiment of the method described in this invention, the loss function of the target detection model constructed by the convolutional neural network is the FocalLoss function, expressed by formula L. FL =-α t (1-p t ) r log(p t ) indicates that α and r are hyperparameters that adjust the sample weight ratio. When the segmentation result is consistent with the true value of the target to be detected, α t Equal to α, if not consistent, α t The value of r is 1-α; the value of r represents the degree of difficulty in misclassifying the sample.

[0038] In one embodiment of the method of the present invention, the white balance processing step further includes:

[0039] The input image is white-balanced using the total reflection algorithm.

[0040] The present invention also provides a vehicle-mounted image-based farmland target detection device for implementing any of the above methods, comprising:

[0041] The object detection model building module is used to construct a convolutional neural network object detection model with an encoding / decoding structure, including:

[0042] The feature extraction module is used to extract features from the original image at multiple levels.

[0043] The color perception module is used to perceive the color of the multi-level original image features;

[0044] The semantic fusion module is used to semantically fuse the multi-level original image features and the color-perceived multi-level original image features.

[0045] The dilated convolution module is used to perform dilated convolution on the semantically fused features;

[0046] The spatial augmentation module is used to perform spatial augmentation on the features after dilated convolution to prevent the segmentation boundary of large targets from overflowing, while small targets have obvious segmentation features.

[0047] The channel fusion module is used to perform channel fusion between the spatially enhanced features and the semantically fused features.

[0048] The upsampling module is used to upsample the features after channel fusion to restore the original image size;

[0049] The prediction module is used to predict the image segmentation result pixel by pixel from the feature map that restores the original image size, so as to obtain the target detection result;

[0050] The training module is used to train the target detection model;

[0051] The object detection module is used to input the image to be detected into the trained object detection model for object detection.

[0052] The present invention also provides an agricultural vehicle, including a chassis, a body located on the chassis, and a vehicle control system, wherein the vehicle control system includes the aforementioned device.

[0053] The present invention also provides a storage medium for storing a computer control program, characterized in that the computer control program is used to execute the steps of any of the above methods.

[0054] Compared to existing technologies, this invention discloses a vehicle-mounted image-based farmland target detection method and device. First, it enhances the color features of the input image. Then, through a cascaded attention perception approach, it strengthens complementary colors to enhance the color features of severely discolored straw in the image through multi-level color feature enhancement (color perception). The multi-level color features are fused and complementary to extract accurate straw feature representations. During the decoding process, a spatial enhancement module improves the distinguishability between straw and the farmland background, optimizes the performance of the straw target detection model, and addresses the problems of small straw targets and ambiguous semantic information in complex environments due to the influence of no-till machine field operation, as well as the susceptibility of straw pixels in images to misjudgment caused by external and internal interference factors.

[0055] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0056] Figure 1A schematic diagram illustrating the use of the gray world algorithm for white balance processing of images in the prior art is shown.

[0057] Figure 2 A schematic diagram illustrating white balance processing of images using the total internal reflection algorithm in the prior art is shown.

[0058] Figure 3 A flowchart illustrating a vehicle-mounted image farmland target detection method according to an embodiment of the present invention is shown.

[0059] Figure 4 An architectural diagram of a target detection model in one embodiment of the present invention is shown.

[0060] Figure 5 A schematic diagram of the structure of a color perception module in one embodiment of the present invention is shown.

[0061] Figure 6 A schematic diagram of the structure of a space enhancement module in one embodiment of the present invention is shown.

[0062] Figure 7 A block diagram illustrating a vehicle-mounted image farmland target detection device according to an embodiment of the present invention is shown.

[0063] Figure 8 A block diagram illustrating a farmland vehicle in one embodiment of the present invention is shown.

[0064] Figure 9 The illustration shows experimental data for straw target detection in one embodiment of the present invention.

[0065] Figure 10 A diagram illustrating the straw target detection effect in one embodiment of the present invention is shown.

[0066] Figure 11 A schematic diagram illustrating the influence of interference factors on the detection results in one embodiment of the present invention is shown.

[0067] Figure 12 This diagram illustrates the target detection effect before and after the detection area is divided in one embodiment of the present invention.

[0068] In the attached figures, the following labels are used:

[0069] S1-S3…Steps

[0070] Steps S11-S18…

[0071] 1…Color perception

[0072] 2…Spatial Enhancement

[0073] 3…Feature Extraction

[0074] 10…Vehicle-mounted image-based farmland target detection device

[0075] 11…Object Detection Model Building Module

[0076] 12…encoder

[0077] 121…Feature Extraction Module

[0078] 122…Color Perception Module

[0079] 123… Semantic Fusion Module

[0080] 13…decoder

[0081] 131… Dilated Convolution Module

[0082] 132… Space Enhancement Module

[0083] 133…Channel Fusion Module

[0084] 134…Upsampling Module

[0085] 135…Prediction Module

[0086] 14… Training Module

[0087] 15…Target Detection Module

[0088] 100…a type of vehicle

[0089] 110…chassis

[0090] 120…body

[0091] 130… Vehicle Control System

[0092] ASPP…Dental Convolution Detailed Implementation

[0093] The following specific embodiments, in conjunction with the accompanying drawings, illustrate the implementation methods disclosed in this invention. Those skilled in the art can understand the advantages and effects of this invention from the content disclosed in this specification. However, the following disclosure is not intended to limit the scope of protection of this invention. Without departing from the spirit of the invention, those skilled in the art can implement this invention with other different embodiments based on different viewpoints and applications.

[0094] For clarity, the figures shown in this invention are simplified schematic diagrams illustrating the basic structure of the invention. Therefore, the structures shown in the figures are not drawn to scale according to the actual shape and size of the implementation. For example, the dimensions of certain components have been enlarged for ease of explanation.

[0095] Furthermore, it should be understood that when a component such as a layer, film, region, or substrate is referred to as being "on" or "connected" to another component, it may be directly on or connected to the other component, or an intermediate component may also be present. Conversely, when a component is referred to as being "directly on" or "directly connected" to another component, no intermediate component exists. As used herein, "connection" can refer to physical and / or electrical connections. Moreover, "electrical connection" or "coupling" can refer to the presence of other components between the two components.

[0096] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this invention pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as they have in the context of the relevant technology and this invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0097] Furthermore, it should be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various components, parts, regions, layers, and / or portions, these components, parts, regions, and / or portions should not be limited by these terms. These terms are used only to distinguish one component, part, region, layer, or portion from another. Therefore, the “first component,” “part,” “region,” “layer,” or “part” discussed below may be referred to as a second component, part, region, layer, or portion without departing from the teachings of this document.

[0098] It should be understood that references to "an embodiment," "embodiment," "example embodiment," etc., in the specification refer to the described embodiment, indicating that it may include specific features, structures, or characteristics, but does not necessarily include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0099] The specification and subsequent claims use certain terms to refer to specific modules, components, or parts. Those skilled in the art will understand that users or manufacturers may use different names or terms to refer to the same module, component, or part. This specification and subsequent claims do not distinguish modules, components, or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Furthermore, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections via other means.

[0100] Furthermore, in the following description and claims, numerous terms will be referenced, which should be defined as having the following meanings. The singular forms “a” and “the” include plural referents, unless the context clearly specifies otherwise. “Optional” or “optionally” indicates that an event or situation subsequently described may or may not occur, and the description includes both the scenario where the event occurs and the scenario where the event does not occur.

[0101] Under lighting conditions, the light intensity sensed by photosensitive electronic components deviates somewhat from the true color of an object. To eliminate this deviation, white balance correction is typically used. This method uses a white object in the image as the color correction benchmark. By restoring the color of the polarized white object, the proportions of blue, green, and red light are altered, thereby restoring other colors in the image. Gray-world algorithms and total internal reflection algorithms are the most commonly used white balance algorithms.

[0102] The Gray-World algorithm is a simple method for white balance correction. This algorithm assumes that given a scene with rich color variations, the average colors of R, G, and B will tend to be equal or at the middle gray level, and that all deviations in the image are caused by the light source. From a physical perspective, in the absence of information about the light source, the average reflectance of objects in nature is a constant in the spectral space. Therefore, using the average RGB values ​​of an image under different lighting conditions can reflect the estimated magnitude of the light source's chromaticity. The Gray-World algorithm applies this principle to visible light images, eliminating the influence of ambient light and obtaining the original scene image. This method can be divided into three steps:

[0103] (1) Calculate the average gray level of a single RGB channel Determine the average gray value

[0104]

[0105] (2) Calculate the gain coefficients of the three RGB channels;

[0106]

[0107] (3) Multiply the RGB values ​​of each pixel in the image by the gain coefficient to obtain the adjusted grayscale value.

[0108]

[0109] Please see Figure 1 In practical applications, due to various image errors and the monochromatic nature of images, it is difficult to meet the assumptions of the gray world algorithm, thus limiting the processing effectiveness of this method. Figure 1 For example, from left to right: original image; first white balance; first image multiplication; second white balance; second image multiplication. Applying the captured straw monitoring image to the grayscale world algorithm, the improvement to the original image after two white balance processes is not significant. After the first multiplication with the original image, the image can still reflect details, but after the second multiplication, the grayscale values ​​of the straw and farmland reach saturation or even overflow. This method not only fails to enhance color features but also increases the difficulty of image processing.

[0110] The total internal reflection algorithm, unlike the gray-world algorithm, only considers the brightest part of the image and does not require a rich color palette. This method assumes that the point with the highest brightness value (R+G+B) in the image is white. If this point is not white, the deviation value at this point is corrected, and this point is used as the reference point for color correction of the entire image. When processing images with rich colors, the brightest point in the image is not necessarily white, which can easily cause color casts. Therefore, the effect is more pronounced when processing images with monochromatic colors. The correction formula of this method is as follows:

[0111]

[0112] Among them, R MAX G MAX B MAX It is the color component value of the point with the maximum brightness, B W G W B W It is the value of the point where the brightness is at its maximum after white balance, which is theoretically 255.

[0113] Figure 2The images processed by the total internal reflection algorithm under white balance are shown from left to right: original image; first white balance; first image multiplication; second white balance; second image multiplication; third white balance; third image multiplication. The results show that the average chromaticity and brightness of non-specular reflective objects in the image are deviated due to the influence of light. The smooth surface of the straw, as the brightest point in the entire image, is used as the reference point for image correction. After three white balance operations, some of the light source color is eliminated, and the contrast between the straw, farmland, and shadows is gradually increased. The pixel values ​​of the fused image tend to stabilize, while retaining the rich details of the original image. Therefore, this method is reliable for removing light interference and enhancing straw features.

[0114] Please see Figure 3 and Figure 4 This invention proposes a method for detecting farmland targets in vehicle-mounted images, comprising the following steps:

[0115] Step S1: Construct a convolutional neural network object detection model with an encoding / decoding structure, including:

[0116] Step S11: Extract multi-level original image features;

[0117] Step S12: Perform color perception on the multi-level original image features;

[0118] Step S13: Semantically fuse the multi-level original image features and the color-perceived multi-level original image features;

[0119] Step S14: Perform dilated convolution on the semantically fused features;

[0120] Step S15: Perform spatial augmentation on the features after dilated convolution to prevent the segmentation boundary of large targets from overflowing, while small targets have obvious segmentation features.

[0121] Step S16: Perform channel fusion on the spatially enhanced features and the semantically fused features;

[0122] Step S17: Upsample the features after channel fusion to restore the original image size;

[0123] Step S18: Predict the image segmentation result pixel by pixel using the feature map that restores the original image size to obtain the target detection result;

[0124] Step S2: Train the target detection model;

[0125] Step S3: Input the image to be detected into the trained target detection model to perform target detection.

[0126] like Figure 4As shown, in one embodiment of the invention, the target detection model is an overall encoder-decoder structure, improved upon Deeplabv3+. The encoding part uses a lightweight residual network structure, forming a dual-branch convolutional neural network as the encoder. This network extracts three levels of original image features and color enhancement features, respectively. A color perception module integrates the features from each level of the original image into the features after white balance processing, enhancing detail information and enriching color features. Specifically, in one embodiment, a 3*3 convolution kernel is used for feature extraction, resulting in 16, 32, and 64 features after the first, third, and fourth levels of feature extraction, respectively. Furthermore, the dual-branch features are input to a semantic fusion module, which superimposes the third-level original image features and the third-level color enhancement features. The resulting feature count is 128, used to integrate the semantic features related to the original image features and color features. Then, in the decoding part, the fused semantic features are subjected to dilated convolution (ASPP) and upsampling to restore the image size step by step. Since multi-scale features lack continuity between pixels due to large-scale expansion, resulting in poor segmentation of small targets, this paper adds a spatial enhancement module before upsampling. The aim is to fully utilize the spatial characteristics of multi-scale features, refine the boundary range of large-scale targets before upsampling, and expand the small target region after upsampling. Finally, the original image resolution is restored through 3×3 convolution and 4x upsampling, and the image segmentation result is predicted pixel-by-pixel using the Softmax function, thus achieving straw target detection.

[0127] In one embodiment of the method of the present invention, the step of extracting multi-level original image features further includes:

[0128] A convolution operation is performed on the input image of each level of original image features, where the input image of the next level of original image features is the original image features of the previous level, and the input image of the first level of original image features is the original input image.

[0129] In one embodiment of the method of the present invention, the color perception of the multi-level original image features is achieved by feature fusion between the multi-level original image features and the color enhancement features corresponding to each of the multi-level original image features.

[0130] In one embodiment of the method of the present invention, the step of extracting the color enhancement features corresponding to each of the multi-level original image features further includes:

[0131] A convolution operation is performed on the input image of each level of color enhancement feature. The input image of the next level of color enhancement feature is the output feature map of the previous level of color enhancement feature after color perception. The input image of the first level of color enhancement feature is the image after white balance processing of the original input image.

[0132] Please see Figure 5In natural ambient light, ground features such as clods of earth, tire tracks, or weeds can also interfere with straw segmentation. Convolutional neural networks can extract semantic information about different objects in an image from shallow to deep during the encoding stage, including points, corners, edges, colors, and textures. The features extracted from the original image tend to focus on edge and texture variations, making straw in tire tracks, shadows, or weeds difficult to encode. However, images processed with white balance reduce the color cast of the light source, and the features focus more on areas with color changes, compensating for color information missed by the original image due to lighting conditions, thus pinpointing the spatial location of the straw. The color perception step and module provided by this invention, through color enhancement and feature fusion, achieve mutual complementarity between two types of features, enabling more accurate determination of straw regions. In one embodiment of the method described in this invention, the color perception step, achieved by feature fusion between the original image features and the corresponding color enhancement features, further includes:

[0133] The original image features and the color enhancement features corresponding to the original image features are subjected to the first set of nonlinear mappings to obtain the nonlinear mapped original image features and the nonlinear mapped color enhancement features.

[0134] The original image features are multiplied by the nonlinear mapping color enhancement features, and then the multiplied product is superimposed on the original image features using the formula. It means that among them This represents the features of the original image after overlay. This represents the original feature map fused with the i-th feature. This represents the nonlinear mapping color enhancement feature fused from the i-th feature;

[0135] The color enhancement feature is multiplied by the original nonlinear mapping feature, and then superimposed on the color enhancement feature using the formula. It means that among them This indicates the color enhancement feature after superposition. This represents the color-enhanced feature map fused from the i-th feature. This represents the nonlinear original image feature fused from the i-th feature;

[0136] Channel fusion is performed on the superimposed original image features and the superimposed color enhancement features, and a second set of nonlinear mapping is applied to the channel fused features.

[0137] Specifically, color perception is achieved through dot product and superposition operations on corresponding pixels in the original image feature map and the color region mask (features after color perception). For two feature maps at the same stage... and Cross-multiplication with the original features can compensate for each other in terms of image color cast and detail blur. Constructing jump connections from the original features and superimposing them with the features after dot product can clearly distinguish the differences between the two types of feature maps.

[0138] because and The eigenvalues ​​range from 0 to 1. and exist and The feature value is enhanced at the corresponding position in the image. When the value is close to 0, the updated feature value is closer to the original feature. When the value is close to 1, the feature value increases, indicating that the position is a region of interest of the residual network. Therefore, fusing the two types of features by channel not only preserves the shallow details of the original image, but also enriches the areas with significant color changes, thus enhancing the color features with the same semantic meaning at the same position.

[0139] In one embodiment of the method of the present invention, the step of performing a first set of nonlinear mappings on the original image features and the color enhancement features corresponding to the original image features further includes:

[0140] Apply a nonlinear mapping to the features of the original image using the formula. express,

[0141] Apply a nonlinear mapping to the color enhancement features using the formula

[0142] In this diagram, f() represents the first convolution operation, BN represents the batch normalization operation, and the nonlinear mapping uses the Sigmoid activation function.

[0143] Before performing the first convolution operation and Pixels are filled on the outer side.

[0144] Specifically, both the original image features and the corresponding color enhancement features have a current size of L. i ×W i The number of channels is T. The purpose of applying a set of nonlinear mappings to the two types of features is to enable the features of the residual network to interact. It should be noted that the first convolution operation is a convolution operation with a 3*3 kernel and a stride of 1, and the first set of nonlinear mappings uses the Sigmoid activation function, but this invention is not limited to this. In addition, before performing the first convolution operation, ... and The outer pixel filling specifically refers to filling pixels with a scale of 1 and a value of 0, but this invention is not limited to this.

[0145] In one embodiment of the method of the present invention, the step of performing a second set of nonlinear mapping on the channel-fused features further includes:

[0146] Where f′() represents the second convolution operation, concat() represents the channel fusion operation, BN represents the batch normalization operation, and the nonlinear mapping uses the ReLU activation function. This represents the multi-level original image features after color perception.

[0147] Specifically, in one embodiment of the present invention, the number of new feature channels formed after channel fusion becomes twice the original, 2T. The second convolution operation is a 1*1 convolution operation with a stride of 1. After feature mapping, the feature size remains L. i ×W i The number of channels is also reduced to T. The activation function used in the second set of nonlinear mappings is the ReLU function, and this is not a limitation.

[0148] Please see Figure 6 In deep convolutional neural networks, more model layers mean deeper feature levels. However, due to the small size of individual straw targets and their relatively small pixel proportion compared to the farmland background, blurred high-level semantic features often hinder the segmentation of small targets. Analysis of the feature extraction patterns of the ASPP module in the Deeplabv3+ encoder reveals that although each feature extraction channel corresponds to semantic information at different scales, the segmentation results after semantic information fusion often tend towards large-scale features. Therefore, this invention provides a spatial enhancement module after the ASPP module. This module uses bilinear interpolation upsampling to strengthen the spatial relationship of targets at different scales, preventing the segmentation boundary of large targets from overflowing while ensuring that small targets have relatively obvious segmentation features.

[0149] In one embodiment of the method of the present invention, the spatial enhancement processing of the features after dilated convolution is achieved by enhancing the spatial positional relationship of targets at different scales through bilinear interpolation upsampling.

[0150] In one embodiment of the method of the present invention, the spatial enhancement processing step of strengthening the spatial positional relationship of targets at different scales through bilinear interpolation upsampling further includes:

[0151] The features after the dilated convolution are subjected to bilinear interpolation upsampling, and then a third set of nonlinear mappings is applied to the features after the bilinear interpolation upsampling to obtain a spatial region mask, which is then expressed using the formula...

[0152] M j =Sigmoid(Up(Down(f″(F)) j )))) represents, where f″() represents the third convolution operation, F j This represents the entire feature map after dilated convolution. Down() represents the downsampling operation, Up() represents the bilinear interpolation upsampling operation, and M... j Indicates a spatial region mask;

[0153] The feature map after dilated convolution is multiplied by the spatial region mask, and the multiplied feature map is then superimposed on the feature map after dilated convolution, using the formula... It means that among them This represents the features after the spatial enhancement process. The feature map represents the multi-scale features of each channel of the dilated convolution output.

[0154] Specifically, for the entire spatial augmentation module, the feature map of each channel of the ASPP multi-scale features... Both are inputs to the module, and the input feature map size is L. j ×W j Then, the spatial region mask M obtained by relearning spatial relationships j The third convolution operation is a convolution operation with a kernel size of 3*3 and a stride of 1. The downsampling operation is a downsampling operation with a stride of 1 and a scale of 2. The third set of nonlinear mappings uses the Sigmoid function. This is not an invention or limitation.

[0155] After bilinear interpolation upsampling, the spatial region mask M j This reflects the weight differences in local regions of the feature map. Weighting the feature map using a spatial region mask can enhance the spatial region features of the feature map. Superimposing it with the original feature map yields a more refined feature representation. in Each pixel in the feature map is determined based on the spatial region mask M. j The weight range is strengthened.

[0156] For straw target detection tasks, straw is usually divided into two categories: straw and background. The formula for the common binary classification cross-entropy loss function is:

[0157] L BCE =-log(p t (5-8)

[0158] Among them, when the segmentation result is consistent with the true value of the straw target, p t The probability of straw detection is equal to p; if they are not consistent, p t It is 1-p.

[0159] Since the number of pixels representing straw targets in the field is relatively low, and each pixel has the same weight during parameter iteration and optimization, the model tends to favor the extraction of features from large targets, which is not conducive to the detection of straw targets that are small. Therefore, this invention adopts the Focal Loss function, which addresses the problem of imbalanced sample classes.

[0160] In one embodiment of the method described in this invention, the loss function of the target detection model constructed by the convolutional neural network is the FocalLoss function, expressed by formula L. FL =-α t (1-p t ) r log(p t ) indicates that α and r are hyperparameters that adjust the sample weight ratio. When the segmentation result is consistent with the true value of the target to be detected, α t Equal to α, if not consistent, α t The value of r is 1-α; the value of r represents the degree of difficulty in misclassifying the sample.

[0161] In one embodiment of the method of the present invention, the white balance processing step further includes:

[0162] The input image is white-balanced using the total reflection algorithm.

[0163] The present invention also provides a vehicle-mounted image-based farmland target detection device for implementing any of the above methods, comprising:

[0164] The object detection model building module is used to construct a convolutional neural network object detection model with an encoding / decoding structure, including:

[0165] The feature extraction module is used to extract features from the original image at multiple levels.

[0166] The color perception module is used to perceive the color of the multi-level original image features;

[0167] The semantic fusion module is used to semantically fuse the multi-level original image features and the color-perceived multi-level original image features.

[0168] The dilated convolution module is used to perform dilated convolution on the semantically fused features;

[0169] The spatial augmentation module is used to perform spatial augmentation on the features after dilated convolution to prevent the segmentation boundary of large targets from overflowing, while small targets have obvious segmentation features.

[0170] The channel fusion module is used to perform channel fusion between the spatially enhanced features and the semantically fused features.

[0171] The upsampling module is used to upsample the features after channel fusion to restore the original image size;

[0172] The prediction module is used to predict the image segmentation result pixel by pixel from the feature map that restores the original image size, so as to obtain the target detection result;

[0173] The training module is used to train the target detection model;

[0174] The object detection module is used to input the image to be detected into the trained object detection model for object detection.

[0175] The present invention also provides an agricultural vehicle, including a chassis, a body located on the chassis, and a vehicle control system, wherein the vehicle control system includes the aforementioned device.

[0176] The present invention also provides a storage medium for storing a computer control program, characterized in that the computer control program is used to execute the steps of any of the above methods.

[0177] The aforementioned computer program may be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0178] Experimental data and related settings of this invention:

[0179] To verify the performance of the straw target detection model proposed in this invention, the experimental data used in this section consists of a total of 1051 images (including 350 test samples) of standard images and images to be processed obtained after data classification in Chapter 4. The straw annotations on the images to be processed referenced the detection region division range in Chapter 4. The images and annotations in the dataset are as follows: Figure 9 As shown in the figure, since each image was acquired under various interference factors, this section divides the image test set into three types according to the main influencing factors to evaluate the model from different perspectives. Due to the prevalent lighting interference in all images, the first type of test data (F1) categorizes all test samples into facing and backlighting based on driving direction and light source position, which is reflected in the image as the influence of light intensity and shadow size, used to compare the overall straw detection accuracy of the model. The second type of test data (F2) selects six types of images based on straw length, furrow depth, and soil clod size to qualitatively analyze the influence of farmland straw morphology and similar ground features on straw detection, and to more comprehensively evaluate the model's detection capability. Furthermore, since straw detection is positively correlated with detection distance, the impact of changes in the detection range caused by camera angle on straw target detection varies. The third type of test data (F3) uses 60 images to analyze the impact of different straw amounts on the division of the detection area.

[0180] The relevant settings used in this invention are as follows: The Nadam parameter optimization algorithm is employed to accelerate model convergence. This method considers first-order and second-order momentum accumulation and dynamically adjusts the learning rate constraint range to avoid the model getting trapped in local optima or prematurely ending training. The model uses the default PyTorch parameter settings: initial learning rate of 0.002, beta1 of 0.9, beta2 of 0.999, epsilon of 1e-8, and a learning rate decay rate of 0.004 after parameter updates. The training cycle is 40 times. The quantitative evaluation method for the model uses accuracy and mean intersection-over-union ratio (MIRU). Because straw is a small and unclear target in the image, the straw annotation error is relatively large. To reduce the error caused by annotation accuracy, the segmentation error of the model is allowed to be within one pixel. That is, if a pixel is judged as true, and there are true values ​​among the nine pixels including the pixel, then the pixel is judged as correct.

[0181] To verify the segmentation performance of each part of the improved model for straw target detection, this section integrates the color perception module and spatial augmentation module into the Deeplabv3+ (DLv3+ or the original model) model step by step. Comparative experiments were conducted on the DLv3+ (original model), DLv3+ / CPM (original model + color perception module), DLv3+ / SEM (original model + spatial augmentation module), and DLv3+ / CPM / SEM (original model + color perception module + spatial augmentation module) models. Parameter training was completed on a straw monitoring image dataset, and the overall model performance was verified using the (F1) test set. The quantitative and qualitative experimental results of straw detection for different models are shown in Tables 1 and 2, respectively. Figure 10 As shown.

[0182] First, the segmentation accuracy and performance of different models were objectively evaluated and analyzed based on the quantitative results of straw detection. Table 1 shows that the proposed DLv3+ / CPM / SEM improved model outperformed other comparative models in both accuracy and overall MIou (Mutual Intersection over Union) evaluation. Compared to DLv3+, models incorporating CPM and SEM modules also slightly improved the accuracy of straw targets. However, CPM increased the influence range of straw targets, leading to more misclassified pixels and a poorer average intersection-over-union (IoU) ratio. When the vehicle is facing the light source, the model with the CPM module can suppress the influence of changes in light intensity; while when facing away from the light source, the model with the SEM module showed more stable accuracy under different shadow occlusions.

[0183] Table 1 Comparison Results of Straw Target Detection Models

[0184]

[0185] like Figure 11As shown, three types of light interference samples and three types of shadow occlusion samples are listed from top to bottom. The original image and the results obtained by various models are displayed from left to right. It can be seen from the figures that DLv3+ achieves basic accuracy in segmenting field straw, but it suffers from problems such as missed detection of small targets and misclassification of community edges for some scattered individual straws or straw communities with overlapping edges. After incorporating two types of improved feature enhancement modules into the model, the segmentation results clearly show that both structures are improved compared to DLv3+. The segmentation map of the CMP module contains some discrete points, which correspond precisely to the scattered small target straws. This is because the CMP module enhances the image color contrast during the encoding stage, highlighting the distribution location of straw targets in the image. However, since the proportion of small target samples is small, they are weakened in higher-order semantics, and the multi-scale feature balancing blurs the lower-level information, making them appear as discrete points in the final result. Compared to DLv3+, the addition of the SEM module inherits the spatial continuity of features at all levels across multiple scales. Feature stacking makes the segmentation boundaries of larger individuals and straw communities with obvious straw features clearer in the resulting image, but smaller, more difficult-to-segment straw targets are missed. Therefore, the fusion model of the CMP and SEM modules complements each other in terms of color and spatial language features, resulting in superior subjective performance in the segmentation image.

[0186] like Figure 12 This section further analyzes potential interference factors in the context of farmland, discussing the impact of straw length, furrow depth, and clod size on straw target detection. Since different interference attributes of the same factor often coexist in the image (e.g., straw of different lengths, furrows of different depths, or clods of different sizes appear simultaneously), accurate quantitative analysis is impossible. Therefore, for each influencing factor, three representative test samples are selected, and the impact of different interference attributes is analyzed through visualization of the straw segmentation map, identifying missed segments (marked with yellow circles) and misclassified segments (marked with green circles).

[0187] This paper presents the results of straw segmentation under different influencing factors using the proposed method, and compares its performance with that of the DLv3+ method. The first set of experimental results comparing straw of different lengths shows that both methods miss smaller straw targets compared to longer straw. The proposed method outperforms DLv3+ in terms of the number of labeled straws and does not exhibit misclassification. This demonstrates that the model proposed in this invention can effectively detect straw in different positions in the field and has a certain ability to distinguish small target straw. The second set shows the results of furrows at different depths. The figures clearly show that if a large amount of straw is scattered in a furrow, the model will mistakenly identify the furrow as longer straw, resulting in strips of varying thickness in the comparison of the two models. The proposed method, due to the boundary constraints imposed by the SEM module, produces thinner strips, thus having a lower impact on straw detection results. The third comparison focuses on the impact of soil clods of different sizes. As shown in the figure, well-shaped soil clods have a relatively small impact on the segmentation results of the two models. However, after deep loosening or deep turning, the texture of the soil clods is similar to the appearance of the straw community. In particular, the straw and soil clods in the loosened soil layer are easily confused, which increases the probability of missed and misclassified segments in the model. In DLv3+, this manifests as misclassification of some regions, but in the method presented in this paper, it manifests as misclassification of pixels. This indicates that the method presented in this paper effectively integrates the CPM and SEM modules, which helps to extract more representative semantic information.

[0188] The previous section analyzed from a principle perspective why distant farmland cannot be used as a sample basis for straw detection, and in Chapter 4, straw detection areas were segmented from the farmland background. However, the impact of segmenting the detection areas on the accuracy of straw coverage rate was not verified from a quantitative perspective. In this experiment, F2 test data is used to analyze the impact of segmenting the detection areas on straw coverage rate from two aspects: changes in detection range caused by camera angle and straw coverage amount. In images with excessively low angles, there are basically no distant farmland. Therefore, the camera angle is divided into three levels: standard, high, and excessively high. The ratio and deviation between the coverage rate calculated by the model and the straw coverage rate in the true image are used as evaluation indicators of the impact of segmenting the detection areas on the results. If the ratio is close to 1, it means that the detection result is closer to the true value, and if it is far from 1, it means that there is a significant missegmentation of straw.

[0189] Table 2 Comparison of results before and after the division of the inspection area.

[0190]

[0191] As shown in Table 2, for the straw target detection task, the average deviation after the detection region is divided in various images is more balanced and better than the result before segmentation. This is because after the detection region is divided, there is no interference from the sky background and distant farmland in the sample images, making the sample images closer to the standard images. However, in the undivided image samples, the higher the camera angle, the larger the proportion of the sky background in the image. During the white balance processing, the sky, which is closer to white, replaces the straw as the benchmark for color correction, causing the color perception module for straw in the model to fail. Therefore, without the detection region being divided, when there is a large amount of straw covering the field, an excessively high angle will lead to an overestimation of the straw coverage rate, while when there is less straw covering, the straw coverage rate will be underestimated.

[0192] like Figure 12 As shown in rows 1 and 2, when the proportion of straw pixels is small, the number of missed areas (marked with yellow circles in the image) in the undivided detection area in column 3 is significantly greater than in the divided image in column 5. This is because straw located in the distance has low resolution and indistinct features, making it easily assimilated by farmland pixels, resulting in lower straw target detection results. When the proportion of straw pixels is large (as shown in rows 3 and 4), the model loss function is generally more biased towards straw targets. In the image in column 3, furrows in the distance without straw cover are assimilated by a large number of straw pixels, resulting in higher straw target detection results. In addition, the occlusion effect of straw elevation on furrows increases with distance, reducing the accuracy of ground feature labeling. Therefore, the division of the detection area can standardize the quality of the image to be detected and plays an important role in improving the accuracy of straw target detection.

[0193] This paper addresses the problem of straw target detection in straw monitoring images affected by external environmental interference. A straw target detection method based on white balance feature enhancement is proposed. This method integrates a color perception module and a spatial enhancement module on the DLv3+ model, utilizing white balance technology to improve the contrast of straw targets in the image, thus maintaining accuracy in straw target detection even under various interference factors. Experiments demonstrate that the proposed DLv3+ / CPM / SEM model outperforms other methods in straw target detection model performance comparison tests, achieving an overall accuracy and average cross-union ratio of 83.79 and 79.45, respectively. Quantitative and qualitative comparisons were conducted under different light source conditions, straw length, furrow depth, and soil clod size interference conditions, and the DLv3+ / CPM / SEM model consistently outperformed the previous DLv3+ model.

[0194] In summary, compared with existing technologies, the vehicle-mounted image farmland target detection method and device disclosed in this invention first enhances the color features of the input image; then, through a cascaded attention perception method, it enhances the color features (color perception) of severely discolored straw in the image by strengthening complementary colors, and fuses the multi-level color features to complement each other, thereby extracting accurate straw feature representations; during the decoding process, the spatial enhancement module improves the distinguishability between straw and farmland background, optimizes the performance of the straw target detection model, and solves the problems of small straw targets and ambiguous semantic information in complex environments due to the influence of no-till machine field operation environment, as well as the easy misjudgment of straw pixels in the image caused by external and internal interference factors.

[0195] The above-disclosed content is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of the patent application of the present invention. Therefore, any equivalent technical changes made based on the contents of the present invention specification and drawings fall within the scope of the patent application of the present invention.

Claims

1. A method for detecting farmland targets in vehicle-mounted images, characterized in that, Includes the following steps: Step S1: Construct a convolutional neural network object detection model with an encoding / decoding structure, including: Step S11: Extract multi-level original image features; Step S12: Perform color perception on the multi-level original image features; Step S13: Semantically fuse the multi-level original image features and the color-perceived multi-level original image features; Step S14: Perform dilated convolution on the semantically fused features; Step S15: Perform spatial augmentation on the features after dilated convolution to prevent the segmentation boundary of large targets from overflowing, while small targets have obvious segmentation features. Step S16: Perform channel fusion on the spatially enhanced features and the semantically fused features; Step S17: Upsample the features after channel fusion to restore the original image size; Step S18: Predict the image segmentation result pixel by pixel using the feature map that restores the original image size to obtain the target detection result; Step S2: Train the target detection model; Step S3: Input the image to be detected into the trained target detection model to perform target detection.

2. The method according to claim 1, characterized in that, The steps for extracting multi-level original image features include: A convolution operation is performed on the input image of each level of original image features, where the input image of the next level of original image features is the original image features of the previous level, and the input image of the first level of original image features is the original input image.

3. The method according to claim 2, characterized in that, The color perception of the multi-level original image features is achieved by feature fusion between the multi-level original image features and the color enhancement features corresponding to each of the multi-level original image features.

4. The method according to claim 3, characterized in that, The steps for extracting the color enhancement features corresponding to each of the multi-level original image features include: A convolution operation is performed on the input image of each level of color enhancement feature. The input image of the next level of color enhancement feature is the output feature map of the previous level of color enhancement feature after color perception. The input image of the first level of color enhancement feature is the image after white balance processing of the original input image.

5. The method according to claim 3 or 4, characterized in that, The color perception step achieved by feature fusion between multi-level original image features and their respective corresponding color enhancement features includes: The original image features and the color enhancement features corresponding to the original image features are subjected to the first set of nonlinear mappings to obtain the nonlinear mapped original image features and the nonlinear mapped color enhancement features. The original image features are multiplied by the nonlinear mapping color enhancement features, and then the multiplied product is superimposed on the original image features using the formula. It means that among them This represents the features of the original image after overlay. Indicates the first The original image feature map fused from the features of each feature. Indicates the first Nonlinear mapping color enhancement features fused from multiple features; The color enhancement feature is multiplied by the original nonlinear mapping feature, and then superimposed on the color enhancement feature using the formula. It means that among them This indicates the color enhancement feature after superposition. Indicates the first A color-enhanced feature map fused from multiple features. Indicates the first Nonlinear original image features fused from multiple features; Channel fusion is performed on the superimposed original image features and the superimposed color enhancement features, and a second set of nonlinear mapping is applied to the channel fused features.

6. The method according to claim 5, characterized in that, The step of performing a first set of nonlinear mappings on the original image features and the color enhancement features corresponding to the original image features includes: Apply a nonlinear mapping to the features of the original image using the formula. express, Apply a nonlinear mapping to the color enhancement features using the formula express, () indicates the first convolution operation. This indicates batch normalization operations; non-linear mappings use... Activation function, where, Before performing the first convolution operation and Pixels are filled on the outer side.

7. The method according to claim 5, characterized in that, The second set of nonlinear mapping steps for the channel-fused features includes: ,in This indicates the second convolution operation. Indicates channel merging operation. This indicates batch normalization operations; non-linear mappings use... Activation function This represents the multi-level original image features after color perception.

8. The method according to claim 1, characterized in that, The spatial enhancement processing of the features after dilated convolution is achieved by strengthening the spatial positional relationships of targets at different scales through bilinear interpolation upsampling.

9. The method according to claim 8, characterized in that, The spatial enhancement processing steps, which enhance the spatial positional relationships of targets at different scales through bilinear interpolation upsampling, include: The features after the dilated convolution are subjected to bilinear interpolation upsampling, and then a third set of nonlinear mappings is applied to the features after the bilinear interpolation upsampling to obtain a spatial region mask, which is then expressed using the formula... It means that among them This indicates the third convolution operation. This represents the complete feature map after dilated convolution. This indicates a downsampling operation. This indicates a bilinear interpolation upsampling operation. Indicates a spatial region mask; The feature map after dilated convolution is multiplied by the spatial region mask, and the multiplied feature map is then superimposed on the feature map after dilated convolution, using the formula... It means that among them This represents the features after the spatial enhancement process. The feature map represents the multi-scale features of each channel of the dilated convolution output.

10. The method according to claim 1, characterized in that, The loss function of the convolutional neural network object detection model is the FocalLoss function, expressed by the formula... It means that among them and These are all hyperparameters that adjust the sample weight ratio. When the segmentation result matches the true value of the target to be detected... equal If they are inconsistent, for ; The value of represents the degree of difficulty in misclassifying the sample.

11. The method according to claim 4, characterized in that, The white balance processing steps include: The input image is white-balanced using the total reflection algorithm.

12. A vehicle-mounted image-based farmland target detection device, used to implement the method as described in any one of claims 1 to 11, characterized in that, include: The object detection model building module is used to construct a convolutional neural network object detection model with an encoding / decoding structure, including: The feature extraction module is used to extract features from the original image at multiple levels. The color perception module is used to perceive the color of the multi-level original image features; The semantic fusion module is used to semantically fuse the multi-level original image features and the color-perceived multi-level original image features. The dilated convolution module is used to perform dilated convolution on the semantically fused features; The spatial augmentation module is used to perform spatial augmentation on the features after dilated convolution to prevent the segmentation boundary of large targets from overflowing, while small targets have obvious segmentation features. The channel fusion module is used to perform channel fusion between the spatially enhanced features and the semantically fused features. The upsampling module is used to upsample the features after channel fusion to restore the original image size; The prediction module is used to predict the image segmentation result pixel by pixel from the feature map that restores the original image size, so as to obtain the target detection result; The training module is used to train the target detection model; The object detection module is used to input the image to be detected into the trained object detection model for object detection.

13. A farm vehicle, comprising a chassis, a body mounted on the chassis, and a vehicle control system, characterized in that, The vehicle control system includes the apparatus as described in claim 12.

14. A storage medium for storing a computer control program, characterized in that, The computer control program is used to perform the steps of the method as described in any one of claims 1 to 11.

Images