Operation environment key element identification method

Abstract

A working environment key element identification method comprises the following steps: S1, collecting and screening working environment scene images to construct a working environment image data set; s2, preprocessing the operation environment image data set constructed in the step S1, and generating a training set, a verification set and a test set; s3, a working environment scene recognition model is constructed, an original DeepLabV3 + model is improved, and a P-Net model is constructed; s4, performing model training on the P-Net model constructed in the S3 by using the training set and the verification set generated in the S2; and S5, performing test evaluation on the P-Net model by using the test set generated in the S2.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a method for identifying key elements of an operating environment. Background Art

[0002] In order to realize the intelligentization of farmland operations, it is necessary to develop specialized intelligent agricultural machinery that can autonomously perform tasks such as plant protection, ditching and fertilization. As a key technology of intelligent agricultural machinery, the perception method of key elements in the operation (such as crops or fruit trees, roads, paths between rows, people and other obstacles) can guide agricultural machinery to perform path planning and autonomous obstacle avoidance, which is of great significance to improving the intelligence of agricultural machinery.

[0003] Traditional computer vision methods are widely used in operating environment perception due to their stability, interpretability and low computational cost, but they are challenging to accurately detect and locate objects in complex operating environments. High-performance deep learning methods, such as Faster R-CNN and YOLO series, excel in object detection and path planning in orchard environments, but they are still limited in capturing 3D shapes and boundary details.

[0004] Pixel-level semantic recognition methods are gradually emerging in orchard information perception, and vehicle-mounted image sensors combined with fine image recognition methods have obvious advantages in vehicle navigation and environmental perception. However, the complex lighting conditions and noise interference in orchards pose challenges to environmental perception, and computational cost and recognition rate are also key issues.

[0005] DeepLabV3 is an advanced pixel-level semantic recognition model. Through the spatial pyramid pooling (ASPP) technology, it can capture multi-scale contextual information in complex scenes and improve the recognition effect. DeepLabV3+ introduces a decoder module on this basis to further enhance the recognition accuracy of boundary areas. However, when identifying key elements of the working environment, DeepLabV3+ still faces some problems: high computational complexity, low recognition speed, and difficulty in meeting the needs of real-time applications; reduced recognition accuracy under complex lighting conditions; and easy to produce false detection and missed detection under noise interference background.

[0006] It is not difficult to see that there are still many problems in the existing technology, and it is necessary to improve the existing method of identifying key elements of the working environment to overcome the defects of the existing technology. Summary of the invention

[0007] Therefore, in order to solve the above problems in the prior art, the present invention proposes a method for identifying key elements of an operating environment.

[0008] The present invention solves the above problems by the following technical means:

[0009] A method for identifying key elements of an operating environment, comprising:

[0010] S1. Collect and screen the working environment scene images to construct a working environment image dataset;

[0011] S2, preprocessing the work environment image dataset constructed in S1 to generate a training set, a validation set and a test set;

[0012] S3, using the feature extraction network MobileNetV2, the selective attention mechanism α module and the lightweight feature extraction mechanism β module to improve the DeeplabV3+ model and build the work environment scene recognition model P-Net;

[0013] S4, using the training set and validation set generated in S2 to perform model training on the P-Net model constructed in S3;

[0014] S5. Use the test set generated in S2 to test, evaluate and optimize the P-Net model.

[0015] Further, the S1 includes:

[0016] S11, collecting images of the working environment, moving along the working path, and using image acquisition equipment to collect RGB images in the moving direction;

[0017] S12, screening and processing the images collected in S11, processing the size of the collected original images to d1*d2 pixels and storing them, so as to construct a work environment image data set.

[0018] Further, the S2 includes:

[0019] S21, data annotation, annotating key elements in the images of the working environment image dataset constructed in S1;

[0020] S22, image enhancement, performing data enhancement processing on the images in the working environment image dataset constructed in S1 to improve the generalization ability of the model;

[0021] S23, data set division, dividing the data set enhanced by S22 into a training set, a validation set and a test set in a ratio of 7:1:2.

[0022] Further, the S3 includes:

[0023] S31. Taking the DeepLabV3+ model as the basic model, the MobileNetv2 feature extraction network is used to replace the Xception feature extraction network in the encoder part of the DeepLabV3+ model to reduce the model size.

[0024] S32, adopt the lightweight feature extraction mechanism β module as the feature extraction algorithm of the multi-scale feature extraction layer DP structure, and use the DP structure to replace the ASPP structure in the DeepLabV3+ model to improve the model recognition speed, and further improve the model's ability to recognize multi-scale targets under noise interference background;

[0025] S33, using a selective attention mechanism α module at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information, further improving the recognition ability of the model under complex lighting conditions;

[0026] S34, using a lightweight feature extraction mechanism β module to replace the standard convolution in the decoder part of the original DeepLabV3+ model to quickly restore the spatial resolution of the feature map and improve the recognition effect;

[0027] S35. The DF loss function is used as the loss function of P-Net to deal with the imbalance problem between target information pixels and background pixels.

[0028] Furthermore, the S32 includes: the DP structure first receives the output feature X of the MobileNetv2 feature extraction network, and uses a lightweight feature extraction mechanism β module with different convolution kernels K and void rates r to extract features, and finally uses global average pooling to capture global context information; wherein the calculation method of the β module is as follows:

[0029]

[0030] In the formula, C in is the number of input channels, Y(h, w, c out ) is the output feature map of the β module. X(h+r·i,w+r·j,c) is the element in the input feature map X after the position adjustment by the dilated convolution, where r is the dilation rate, and i and j are the spatial indices of the convolution kernel. W dw (i, j, c) is a depthwise convolution kernel of size K×K, and there is a separate convolution kernel for each input channel c. pw (c,c out ) is the convolution kernel of point-by-point convolution, with a size of 1×1×C in ×C out .

[0031] Further, the S33 includes: the selective attention mechanism α module performs the input feature map Processing is performed to assign a weight to each channel, where H×W×C represents the height, width, and number of channels of the feature map respectively;

[0032] First, global average pooling is performed on each channel to obtain a C-dimensional vector. Then, the pooled feature vector is processed by a one-dimensional convolution operation Conv1D(·) to generate a channel vector. Then, the convolution result is normalized using the Sigmoid activation function σ(·) to obtain the final channel weight. Finally, the weight is applied channel by channel to the original input feature map X to obtain the weighted feature map X′, thereby selectively enhancing the features of important channels and suppressing unimportant channel information.

[0033]

[0034] In the formula, X ijc Represents X ijc The value at position (i, j) and channel c.

[0035] Furthermore, the S34 includes: using the lightweight feature extraction mechanism β module to replace the standard convolution operation of the decoder part of the original DeepLabV3+ model; during the feature extraction process, the computational amount of the β module is 1 / K of the computational amount of the standard convolution. 2 , where K is the size of the convolution kernel; the calculation formulas are as follows:

[0036] Calculation Amount β =C in ×K 2 ×H×W+C in ×C out ×H×W

[0037] Calculation Amount 标准卷积 =C in ×C out ×K 2 ×H×W

[0038] Calculation Amount β / Calculation 标准卷积 =1 / K 2 +1 / C out ≈1 / K 2 .

[0039] Further, the S35 includes: using the DF loss function as the loss function of the P-Net to handle the imbalance problem between the target information pixels and the background pixels;

[0040] First, by optimizing the overlap between the target area and the prediction area, the segmentation accuracy of the target area is enhanced; secondly, by weighting the contribution of the background area, the interference of background pixels on the training process is reduced, making the model more focused on the learning of the target area; the DF calculation formula is as follows:

[0041]

[0042] In the formula, ∈ is a small constant, N is the total number of samples, and are the true label and prediction result of the i-th sample respectively.

[0043] Further, the S4 includes:

[0044] S41, configure training hyperparameters, input the image resolution of the training set as d1*d2, the batch size as B, and the number of iterations as E rounds;

[0045] S42, start the training model, calculate the model loss value and accuracy value in each training iteration, and update the model weight;

[0046] S43. During the training process, the P-Net model regularly outputs training indicators, including loss values ​​and accuracy indicators, and verifies them on the validation set to obtain the performance indicators of the model.

[0047] Furthermore, the test evaluation in S5 includes the following indicators:

[0048] The pixel accuracy PA is calculated as follows: P ii is the total number of pixels belonging to class i and predicted as class i, P ij is the total number of pixels belonging to class i and predicted to be class j, and K+1 is the number of classes;

[0049] The average pixel accuracy MPA is calculated as follows: P ii is the total number of pixels belonging to class i and predicted as class i, P ij is the total number of pixels belonging to class i and predicted to be class j, and K+1 is the number of classes;

[0050] The average intersection-over-union ratio MIOU is calculated as follows: P ii is the total number of pixels belonging to class i and predicted as class i, P ji is the total number of pixels that belong to class j but are predicted to be class i, and K+1 is the number of classes.

[0051] The model optimization in S5 includes:

[0052] First, based on the evaluation results, analyze the performance of the model in each category, especially focusing on areas with low target detection and segmentation accuracy; second, dynamically adjust the weights in the loss function to enhance the optimization effect on difficult samples and small targets; then, prevent overfitting and improve the generalization ability of the model by adopting a learning rate scheduler and early stopping strategy; finally, fine-tune the structure or hyperparameters of the P-Net model based on the evaluation results to ensure the optimal accuracy of key element recognition in the operating environment and improve the practical application performance and stability of the model.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention significantly reduces the model size and improves the recognition speed by replacing the Xception feature extraction network of the encoder part of DeepLabV3+ with the MobileNetv2 feature extraction network. A lightweight feature extraction mechanism β module is used as the feature extraction algorithm of the multi-scale feature extraction layer DP structure, and the DP structure is used to replace the ASPP structure in the DeepLabV3+ model to improve the model recognition speed, and further improve the model's recognition ability for multi-scale targets under noise interference background. The selective attention mechanism α module is used at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information, so as to further improve the model's recognition ability under complex lighting conditions. The standard convolution of the decoder part in the original DeepLabV3+ model is replaced by a lightweight feature extraction mechanism β module to quickly restore the spatial resolution of the feature map and improve the recognition effect. Finally, for the constructed P-Net model, the DF loss function is used as the loss function to deal with the imbalance problem between target information pixels and background pixels.

[0055] Compared with recognition models such as DeepLabV3+, U-Net, PSPNet, Segformer and HRNet, P-Net has higher recognition accuracy and speed when detecting working environment images, and can well identify key elements in the working environment. In addition, the P-Net model has the advantages of small memory usage, high accuracy and fast speed, and can well support the working machine to perform autonomous navigation and obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1This is the original network structure diagram of DeepLabV3+ described in the present invention;

[0058] Figure 2 It is a network structure diagram of the P-Net described in the present invention;

[0059] Figure 3 This is a schematic diagram of the DP structure of the present invention; Figure 4 This is a schematic diagram of the α module of the present invention;

[0060] Figure 5 Schematic diagram of the comparison between the standard convolution and β module structures of the present invention;

[0061] Figure 6 This is a training result diagram of the P-Net described in the present invention;

[0062] Figure 7 This is a diagram showing the recognition effect of the P-Net of the present invention on the pomelo orchard images under different lighting conditions;

[0063] Figure 8 This is a diagram showing the recognition effect of the P-Net described in the present invention on pomelo orchard images with different degrees of noise. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be pointed out that the described implementation examples are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Example

[0066] Taking the identification of key elements in an orchard operating environment as an example, the method of the present invention is described in detail below: A method for identifying key elements in an operating environment comprises:

[0067] S1, collecting and screening the working environment scene images to construct a working environment image data set; that is, collecting and screening the orchard scene images to construct an orchard image data set; preferably, S1 includes:

[0068] S11, working environment image acquisition, moving along the working path, using the image acquisition device to acquire the RGB image of the moving direction; that is, orchard image acquisition, moving along the path between the fruit tree rows in the orchard, using the image acquisition device to acquire the RGB image of the moving direction; it should be noted that, the present invention preferably allows the working vehicle to move along the path between the fruit tree rows in the orchard, and at the same time calls the camera installed on the working vehicle to acquire the RGB image in front of the working vehicle.

[0069] S12, filtering and processing the images collected in the S11, processing the size of the collected original images to d1*d2 pixels and storing them to construct the working environment image data set. It should be noted that the d1*d2 pixels can be selected to be adapted to the actual application. That is, filtering and processing the images collected in the S11, processing the size of the collected original images to 576×324 pixels and storing them to construct the orchard image data set. The size of the RGB image collected by the camera in front of the working vehicle is generally 2208×1242 pixels, so it needs to be scaled to 576×324 pixels to improve the model detection speed. Preferably, the present invention preferably compresses the original image size to 576×324 pixels and stores it. Because the larger the image size, the slower the detection speed, and the smaller the size, the faster the detection speed, but the recognition accuracy may be reduced. Moreover, if it is not proportionally scaled, it will cause image distortion, so the present invention preferably adopts proportionally scaled images.

[0070] S2, preprocessing the working environment image dataset constructed in S1 to generate a training set, a verification set and a test set; that is, preprocessing the orchard image dataset constructed in S1 to generate a training set, a verification set and a test set; preferably, S2 includes:

[0071] S21, data annotation, annotating the key elements in the pictures in the working environment image dataset constructed in the S1; that is, annotating the pictures in the orchard image dataset constructed in the S1; preferably, the present invention uses the open source software Labelme to annotate the orchard image dataset constructed in the S12, and selects the json annotation format to mark the fruit trees, roads, paths between fruit tree rows, people and obstacles in the pictures. The advantage of the Json format is that structured data is easy to express and parse the object information in the image, and widely supports various deep learning frameworks, which is convenient for subsequent model training and processing. The model proposed in the present invention supports multiple dataset formats, the most common of which are COCO, Pascal VOC and TFRecord formats. The specific choice may depend on the type, size and project requirements of the dataset.

[0072] S22, image enhancement, data enhancement processing is performed on the pictures in the working environment image dataset constructed in the S1 to improve the generalization ability of the model; that is, data enhancement is performed on the pictures in the orchard image dataset constructed in the S1 to improve the generalization ability of the model; it should be noted that the method of data enhancement of the pictures in the orchard image dataset includes but is not limited to: rotation, translation, flipping, noise addition and brightness adjustment to alleviate model overfitting and improve the generalization ability of the model. It should be noted that for image enhancement, in addition to rotation, translation, flipping, noise addition and brightness adjustment, scaling, shearing, color jittering, blurring and sharpening, random cropping, channel exchange, hybrid enhancement, occlusion enhancement, elastic deformation, gamma correction and other methods can also be used. These enhancement methods help to improve the adaptability of the model to different scenes and changes, prevent overfitting, and enhance generalization ability.

[0073] S23, data set division, divide the data set after the enhancement in S22 into a training set, a validation set and a test set in a ratio of 7:1:2.

[0074] S3. Use the feature extraction network MobileNetV2, the selective attention mechanism α module and the lightweight feature extraction mechanism β module to improve the DeeplabV3+ model and build the working environment scene recognition model P-Net; that is, build an orchard scene recognition model and build the original DeepLabV3+ model into a P-Net model.

[0075] It should be noted that the original DeepLabV3+ model includes an encoder part and a decoder part:

[0076] In the encoder part, DeepLabV3+ uses the Xception network in the encoder part. The Xception network can effectively capture the spatial features and contextual information of the image and provide high-quality feature maps for subsequent feature processing. However, due to the large number of layers, the overall computational complexity is still high. At the end of the encoder, DeepLabV3+ introduces the spatial pyramid pooling ASPP module, which captures multi-scale contextual information through hole convolutions of different scales to enhance the model's perception ability at different scales, but multi-scale hole convolutions increase the computational cost.

[0077] In the decoder part, first, the decoder performs channel dimension reduction on the low-level feature map using a 1×1 standard convolution, reducing the number of channels from 256 to 48. Next, the feature map from ASPP is interpolated and upsampled to make it the same size as the low-level feature map after dimension reduction. This step ensures that the feature maps are spatially aligned, which facilitates subsequent fusion processing. The low-level feature map after channel dimension reduction is concatenated with the feature map obtained by interpolation upsampling, and sent to a set of 3×3 standard convolution blocks for further processing; through feature fusion, the decoder can combine low-level and high-level features, but the noise in the low-level and high-level features may affect the recognition accuracy. Finally, linear interpolation upsampling is performed to restore the feature map to the same resolution as the original image, thereby generating the final prediction image.

[0078] Preferably, S3 includes:

[0079] S31. Taking the DeepLabV3+ model as the base model, the MobileNetv2 feature extraction network is used to replace the Xception feature extraction network in the encoder part of the DeepLabV3+ model to reduce the model size; that is, the MobileNetv2 feature extraction network is used to replace the Xception feature extraction network in the encoder part of the DeepLabV3+ model to improve the feature extraction efficiency. It should be noted that because MobileNetv2 is a lightweight network, it is designed with deep separable convolution and inverted residual structure, which significantly reduces the number of parameters and the amount of calculation, while maintaining good feature extraction capabilities, and can complete feature extraction with lower computing resources and latency. In contrast, the Xception network is more complex, using more residual connections and deep architectures, which brings about a larger number of parameters and calculations.

[0080] S32. Use a lightweight feature extraction mechanism β module as the feature extraction algorithm of the multi-scale feature extraction layer DP structure, and use the DP structure to replace the ASPP structure in the DeepLabV3+ model to improve the model recognition speed, and further improve the model's ability to recognize multi-scale targets under noise interference. As a preferred method, the DP structure first receives the output feature X of the MobileNetv2 feature extraction network, and uses a lightweight feature extraction mechanism β module with different convolution kernels K and void rates r for feature extraction, and finally uses global average pooling to capture global context information. The calculation method of the β module is as follows:

[0081]

[0082] In the formula, C in is the number of input channels, Y(h, w, c out) is the output feature map of the β module. X(h+r·i,w+r·j,c) is the element in the input feature map X after the position adjustment by the dilated convolution, where r is the dilation rate, and i and j are the spatial indices of the convolution kernel. W dw (i, j, c) is a depthwise convolution kernel of size K×K, and there is a separate convolution kernel for each input channel c. pw (c,c out ) is the convolution kernel of point-by-point convolution, with a size of 1×1×C in ×C out .

[0083] S33, respectively using the selective attention mechanism α module at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information, further improving the recognition ability of the model under complex lighting conditions; that is, respectively using the selective attention mechanism α module at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information under different lighting conditions, so as to further improve the recognition ability of the model under complex lighting conditions. Preferably, the selective attention mechanism α module performs the input feature map Processing is performed to assign a weight to each channel, where H×W×C represents the height, width, and number of channels of the feature map. First, global average pooling is performed on each channel to obtain a C-dimensional vector; then, the pooled feature vector is processed through a one-dimensional convolution operation Conv1D(·) to generate a channel vector; then, the convolution result is normalized using the Sigmoid activation function σ(·) to obtain the final channel weight; finally, the weight is applied channel by channel to the original input feature map X to obtain the weighted feature map X′, thereby selectively enhancing the features of important channels and suppressing unimportant channel information;

[0084]

[0085] In the formula, X ijc Represents X ijc The value at position (i, j) and channel c.

[0086] S34, using a lightweight feature extraction mechanism β module to replace the standard convolution of the decoder part of the original DeepLabV3+ model to quickly restore the spatial resolution of the feature map and improve the recognition effect. Preferably, the lightweight feature extraction mechanism β module is used to replace the standard convolution operation of the decoder part of the original DeepLabV3+ model; during the feature extraction process, the calculation amount of the β module is 1 / K of the standard convolution calculation amount. 2 ; Due to C out Very large, so the computational effort of the β module is only 1 / K of the computational effort of the standard convolution2 , where K is the size of the convolution kernel; the calculation formulas are as follows:

[0087] Calculation Amount β =C in ×K 2 ×H×W+C in ×C out ×H×W

[0088] Calculation Amount 标准卷积 =C in ×C out ×K 2 ×H×W

[0089] Calculation Amount β / Calculation 标准卷积 =1 / K 2 +1 / C out ≈1 / K 2 .

[0090] S35. Use DF loss function as the loss function of P-Net to deal with the imbalance between target information pixels and background pixels. First, by optimizing the overlap between the target area and the prediction area, the segmentation accuracy of the target area is enhanced to avoid the background area from having too much influence on the total loss, especially when there are fewer pixels in the target area. Secondly, by weighting the contribution of the background area, the interference of background pixels on the training process is reduced, so that the model can focus more on the learning of the target area. The DF calculation formula is as follows:

[0091]

[0092] In the formula, ∈ is a small constant, N is the total number of samples, and are the true label and prediction result of the i-th sample respectively.

[0093] It should be noted that in the P-Net model:

[0094] In the encoder part, the MobileNetv2 feature extraction network is used to replace the Xception feature extraction network in the encoder part of the DeepLabV3+ model to reduce the model size; a lightweight feature extraction mechanism β module is used as the feature extraction algorithm of the multi-scale feature extraction layer DP structure, and the DP structure is used to replace the ASPP structure in the DeepLabV3+ model to improve the model recognition speed, and further improve the model's recognition ability for multi-scale targets under noise interference background;. In addition, the selective attention mechanism α module is used at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information, so as to further improve the model's recognition ability under complex lighting conditions. In the decoder part, the lightweight feature extraction mechanism β module is used to replace the standard convolution of the decoder part in the original DeepLabV3+ model to quickly restore the spatial resolution of the feature map and improve the recognition effect. In addition, the DF loss function is used as the loss function of P-Net to deal with the imbalance problem between target information pixels and background pixels. S4, use the training set and verification set generated in S2 to train the P-Net model constructed in S3; preferably, S4 includes:

[0095] S41, configuring training hyper parameters, inputting the image resolution of the training set as d1*d2, the batch size as B, and the number of iterations as E rounds; that is, configuring training hyper parameters, inputting the image resolution of the training set as 576×324, the batch size as 8, and the number of iterations as 500 rounds;

[0096] S42, start the training model, calculate the model loss value and accuracy value in each training iteration, and update the model weight;

[0097] S43. During the training process, the P-Net model regularly outputs training indicators, including loss values ​​and accuracy indicators, and verifies them on the validation set to obtain the performance indicators of the model.

[0098] S5. Use the test set generated in S2 to test, evaluate and optimize the P-Net model.

[0099] Preferably, the test evaluation in S5 includes the following indicators:

[0100] Pixel accuracy, which is calculated as: P ii is the total number of pixels belonging to class i and predicted as class i, P ij is the total number of pixels belonging to class i and predicted to be class j, and K+1 is the number of classes;

[0101] The average pixel accuracy MPA is calculated as follows: P iiis the total number of pixels belonging to class i and predicted as class i, P ij is the total number of pixels belonging to class i and predicted to be class j, and K+1 is the number of classes;

[0102] The average intersection-over-union ratio MIOU is calculated as follows: P ii is the total number of pixels belonging to class i and predicted as class i, P ji is the total number of pixels that belong to class j but are predicted to be class i, and K+1 is the number of classes.

[0103] Preferably, the model optimization in S5 includes: first, analyzing the performance of the model in each category based on the evaluation results, paying particular attention to areas with low target detection and segmentation accuracy; second, dynamically adjusting the weights in the loss function to enhance the optimization effect on difficult samples and small targets; then, preventing overfitting and improving the generalization ability of the model by adopting a learning rate scheduler and an early stopping strategy; finally, fine-tuning the structure or hyperparameters of the P-Net model based on the evaluation results to ensure that the key element recognition accuracy in the operating environment is optimized and to improve the practical application performance and stability of the model.

[0104] To make the present invention easier to understand, the method of the present invention is further described below by taking a pomelo orchard as an example:

[0105] S1. Collect and screen orchard scene images to construct an orchard image dataset; that is, use a camera to collect pomelo orchard scene images to construct a pomelo orchard image dataset.

[0106] S11, orchard image acquisition, moving along the path between the fruit tree rows in the orchard, using the image acquisition device to collect the RGB image of the moving direction; controlling the work vehicle to move along the path between the fruit tree rows in the honey pomelo orchard, and at the same time calling the camera installed on the work vehicle to collect the RGB image in front of the work vehicle.

[0107] S12, filtering and processing the images collected in S11, processing the size of the original images collected to 576×324 pixels and storing them to construct an orchard image dataset. The collected images are filtered and processed, and the original image size of 2208×1242 pixels is scaled to 576×324 pixels to improve the model detection speed.

[0108] S2. Preprocessing the orchard image dataset constructed in S1 to generate a training set, a validation set, and a test set; that is, preprocessing the honey pomelo orchard dataset includes labeling and image enhancement, and dividing it into a training set, a validation set, and a test set.

[0109] S21, data labeling, labeling the pictures in the orchard image dataset constructed in S1; that is, using the open source software Labelme to label the pictures, selecting the json labeling format, and marking the pomelo trees, roads, paths between fruit trees, people, signboards and street lights in the pictures.

[0110] S22, image enhancement, performing data enhancement on the images in the orchard image dataset constructed in S1 to improve the generalization ability of the model; that is, performing data enhancement on the images, including rotation, translation, flipping, noise addition and brightness adjustment to alleviate model overfitting and improve the generalization ability of the model.

[0111] S23, data set division, dividing the data set enhanced by S22 into a training set, a validation set and a test set in a ratio of 7:1:2.

[0112] S3. Construct an orchard scene recognition model and convert the original DeepLabV3+ model into a P-Net model; that is, construct a pomelo orchard scene recognition model, including improving the original DeepLabV3+ model to obtain P-Net.

[0113] like Figure 1 As shown, the network structure of the original DeepLabV3+ includes an encoder part and a decoder part.

[0114] The feature extraction backbone network of the encoder is Xception. The feature map output by Xception is divided into two parts: one is the feature map of the last layer of convolution output, and the other is the feature map of the intermediate low-level features; the first part of the Xception output is sent to the ASPP module, and the second part is sent to the decoder. The ASPP module accepts the first part of the output of Xception as input, and uses four different dilation rate hole convolution blocks (including convolution, BN, activation layer) and a global average pooling block (including pooling, convolution, BN, activation layer) to obtain a total of five groups of feature maps. After splicing them together, they pass through a 1×1 convolution block (including convolution, BN, activation, dropout layer), and finally sent to the decoder.

[0115] The decoder first convolves the low-level features from the shallow layer of the encoder with a 1×1 convolution, and then fuses them with the output features of the encoder. The low-level features usually retain more spatial detail information, such as edges and textures. Then, the feature map output by the encoder is upsampled by 4 times to restore the low-resolution feature map to a higher resolution. The upsampled feature map is concatenated with the low-level feature map in the channel dimension to form a new feature map. The new feature map is processed through multiple convolutional layers to further fuse and refine the features. The final feature map is restored to the original resolution of the input image through the upsampling layer again. A 3×3 convolutional layer is used to convert the feature map to the same number of channels as the number of categories, generating a pixel-level classification probability map. The category of each pixel is determined by the classification probability map, and the final output is a recognition map of the same size as the input image, with each pixel marked as a specific category.

[0116] The encoder and decoder of DeepLabV3+ are improved respectively to obtain the P-Net model, as shown in Figure 2 Specific improvements include:

[0117] Improved encoder part:

[0118] 1) Taking the DeepLabV3+ model as the basic model, the MobileNetv2 feature extraction network is used to replace the Xception feature extraction network in the encoder part of the DeepLabV3+ model to reduce the model size; Figure 3 As shown in the figure, a lightweight feature extraction mechanism β module is used as the feature extraction algorithm of the multi-scale feature extraction layer DP structure, and the DP structure is used to replace the ASPP structure in the DeepLabV3+ model to improve the model recognition speed and further improve the model's ability to recognize multi-scale targets under noise interference; Figure 4 As shown, a selective attention mechanism α module is used at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information, so as to further improve the recognition ability of the model under complex lighting conditions.

[0119] 2) Improve the decoder part:

[0120] like Figure 5 As shown in the figure, a lightweight feature extraction mechanism β module is used to replace the standard convolution in the decoder part of the original DeepLabV3+ model to quickly restore the spatial resolution of the feature map and improve the recognition effect. The DF loss function is used as the loss function of P-Net to deal with the imbalance between target information pixels and background pixels, ensuring that the model has robust performance.

[0121] S4, using the training set and validation set generated in S2 to perform model training on the P-Net model constructed in S3; that is, using the honey pomelo orchard dataset to perform model training on the P-Net.

[0122] S41, configure training hyperparameters, input the image resolution of the training set as 576×324, the batch size as 8, and the number of iterations as 500 rounds;

[0123] S42, start the training model, calculate the model loss value and accuracy value in each training iteration, and update the model weight;

[0124] S43. During the training process, the P-Net model regularly outputs training indicators, including loss values ​​and accuracy indicators, and verifies them on the validation set to obtain the performance indicators of the model. Figure 6 The figure shows the mIoU curve and loss curve obtained by P-Net on the validation set. The mIoU value of P-Net quickly rose to 70% after the start of training, and began to converge rapidly around 40 rounds, and finally reached about 89% after multiple iterations. In addition, the loss value of P-Net also dropped rapidly after the start of training, and began to converge around 0.25, and finally converged to below 0.18 after 500 iterations of model training. The results show that the training process of P-Net is stable, and there is no sign of overfitting or underfitting.

[0125] S5. Use the test set generated in S2 to test and evaluate the P-Net model.

[0126] Ablation experiments were conducted on the test set to verify the effects of various improvements on P-Net. The results are shown in Table 1. The mIoU and mPA of DeepLabV3+ on the test set reached 80.8% and 84%, respectively. However, with the integration of MobileNetv2, DP structure and α module, the performance of DeepLabV3+ was significantly enhanced. The mIoU increased to 86.2%, 88.4% and 89.3%, respectively, and the mPA increased to 90.8%, 93.8% and 95.1%, respectively. It is worth noting that P-Net exhibits excellent recognition accuracy, with an increase of 8.5% in mIoU and 11.1% in mPA compared to the original DeepLabV3+. In addition, P-Net shows advantages in computational complexity and model size. Compared with DeepLabV3+, the number of parameters, FLOPs and volume are reduced by 94.8%, 94.3% and 94.6%, respectively, while achieving a detection speed of 32FPS.

[0127] Table 1

[0128]

[0129] The test set was used to test the recognition effect of P-Net on orchard images under different lighting conditions. Figure 7 As shown in Table 2, compared with U-Net, PSPNet, Segformer and HRNet, the mIoU value of P-Net is improved by 2.3%, 5.1%, 3.6% and 2.7%, and the mPA value is improved by 1.5%, 4.9%, 1.3% and 2.7%, respectively. In addition, while maintaining a lightweight volume, P-Net achieves a detection speed of 32FPS. The results show that P-Net exhibits good robustness and generalization ability in the face of different lighting scenes.

[0130] Table 2

[0131] Model Number of pictures mIoU / % mPA / % Parameter quantity / M FLOPs / G Volume / MB Speed / FPS P-Net 424 89.8 95.2 2.87 9.93 11.2 32 U-Net 424 87.5 93.7 24 159 94.9 27 PSPNet 424 84.7 90.3 2.37 2.20 9.3 23.2 Segformer 424 86.2 93.9 3.72 4.88 14.2 17.2 HRNet 424 87.1 92.5 9.64 13.49 37.5 20.5

[0132] The test set is used to test the recognition effect of P-Net on orchard images with different degrees of noise. The recognition results of P-Net on orchard images with different degrees of noise are shown in the figure below. Figure 8 and Table 3. The recognition results show that P-Net performs well in recognizing images with slight noise, with a PA of more than 91.3% for each perceptual element, and mIoU and mPA reaching 90.2% and 95.4% respectively. When P-Net recognizes images with moderate noise and severe noise, the mIoU is 86.2% and 84.7% respectively, and the mPA is 93.2% and 90.9% respectively. Compared with the preferred U-Net, P-Net has 1.6%, 1.8% and 6.6% higher mIoU and 6.4%, 9.1% and 11.1% higher mPA respectively when recognizing images with slight noise, moderate noise and severe noise. The results show that P-Net has good recognition ability for orchard images with different degrees of noise.

[0133] Table 3

[0134]

[0135] The present invention significantly reduces the model size and improves the recognition speed by replacing the encoder part of DeepLabV3+ from the Xception feature extraction network to the MobileNetv2 feature extraction network. A lightweight feature extraction mechanism β module is used as the feature extraction algorithm of the multi-scale feature extraction layer DP structure, and the DP structure is used to replace the ASPP structure in the DeepLabV3+ model to improve the model recognition speed, and further improve the model's recognition ability for multi-scale targets under noise interference background. The selective attention mechanism α module is used at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information, further improving the model's recognition ability under complex lighting conditions. In the encoder part, a lightweight feature extraction mechanism β module is used to replace the standard convolution of the decoder part in the original DeepLabV3+ model to quickly restore the spatial resolution of the feature map and improve the recognition effect. Finally, for the constructed P-Net model, the DF loss function is used as the loss function to deal with the imbalance problem between target information pixels and background pixels.

[0136] Compared with recognition models such as DeepLabV3+, U-Net, PSPNet, Segformer and HRNet, P-Net has higher recognition accuracy and speed in the recognition of orchard images. It can well identify key elements in the orchard (such as fruit trees, roads, paths between fruit tree rows, people and other obstacles). In addition, the P-Net model has the advantages of small memory usage, high accuracy and fast speed, and can well support the autonomous navigation and obstacle avoidance of intelligent agricultural machinery in the orchard.

[0137] References to "one embodiment", "another embodiment", "embodiment", "preferred embodiment", etc. in this specification refer to specific features, structures or characteristics described in conjunction with the example included in at least one embodiment generally described in this application. The appearance of the same expression in multiple places in the specification does not necessarily refer to the same embodiment. Further, when a specific feature, structure or characteristic is described in conjunction with any embodiment, it is claimed that the realization of such feature, structure or characteristic in conjunction with other embodiments also falls within the present invention. Although the present invention is described herein with reference to a number of illustrative examples of the present invention, it should be understood that those skilled in the art can design many other modifications and implementations that fall within the scope and spirit of the principles disclosed in this application. More specifically, within the scope of the present application disclosure, drawings and claims, in addition to the deformation and improvement that can be made to the components and / or layout of the subject combination layout, other uses will also be obvious to those skilled in the art.

Claims

1. A method for identifying key elements of a working environment, characterized in that: include: S1. Collect and screen the working environment scene images to construct a working environment image dataset; S2, preprocessing the work environment image dataset constructed in S1 to generate a training set, a validation set and a test set; S3, using the feature extraction network MobileNetV2, the selective attention mechanism α module and the lightweight feature extraction mechanism β module to improve the DeeplabV3+ model and build the work environment scene recognition model P-Net; S4, using the training set and validation set generated in S2 to perform model training on the P-Net model constructed in S3; S5. Use the test set generated in S2 to test, evaluate and optimize the P-Net model.

2. The method for identifying key elements of an operating environment according to claim 1, characterized in that: The S1 includes: S11, collecting images of the working environment, moving along the working path, and using image acquisition equipment to collect RGB images in the moving direction; S12, screening and processing the images collected in S11, processing the size of the collected original images to d1*d2 pixels and storing them, so as to construct a work environment image data set.

3. The method for identifying key elements of an operating environment according to claim 1, characterized in that: The S2 includes: S21, data annotation, annotating key elements in the images of the working environment image dataset constructed in S1; S22, image enhancement, performing data enhancement processing on the images in the working environment image dataset constructed in S1 to improve the generalization ability of the model; S23, data set division, dividing the data set enhanced by S22 into a training set, a validation set and a test set in a ratio of 7:1:

2.

4. The method for identifying key elements of an operating environment according to claim 1, characterized in that: The S3 includes: S31. Taking the DeepLabV3+ model as the basic model, the MobileNetv2 feature extraction network is used to replace the Xception feature extraction network in the encoder part of the DeepLabV3+ model to reduce the model size. S32, adopt the lightweight feature extraction mechanism β module as the feature extraction algorithm of the multi-scale feature extraction layer DP structure, and use the DP structure to replace the ASPP structure in the DeepLabV3+ model to improve the model recognition speed, and further improve the model's ability to recognize multi-scale targets under noise interference background; S33, using a selective attention mechanism α module at the output end of the MobileNetv2 feature extraction network and the output end of the DP structure to focus on extracting key information, further improving the recognition ability of the model under complex lighting conditions; S34, using a lightweight feature extraction mechanism β module to replace the standard convolution in the decoder part of the original DeepLabV3+ model to quickly restore the spatial resolution of the feature map and improve the recognition effect; S35. The DF loss function is used as the loss function of P-Net to deal with the imbalance problem between target information pixels and background pixels.

5. The method for identifying key elements of an operating environment according to claim 4, characterized in that: The S32 includes: the DP structure first receives the output feature X of the MobileNetv2 feature extraction network, and uses a lightweight feature extraction mechanism β module with different convolution kernels K and void rates r to extract features, and finally uses global average pooling to capture global context information; wherein the calculation method of the β module is as follows: In the formula, C in is the number of input channels, Y(h, w, c out ) is the output feature map of the β module. X(h+r·i,w+r·j,c) is the element in the input feature map X after the position adjustment by the dilated convolution, where r is the dilation rate, and i and j are the spatial indices of the convolution kernel. W dw (i, j, c) is a depthwise convolution kernel of size K×K, and there is a separate convolution kernel for each input channel c. pw (c,c out ) is the convolution kernel of point-by-point convolution, with a size of 1×1×C in ×C out .

6. The method for identifying key elements of an operating environment according to claim 5, characterized in that: The S33 includes: the selective attention mechanism α module is input through the feature map Processing is performed to assign a weight to each channel, where H×W×C represents the height, width, and number of channels of the feature map respectively; First, global average pooling is performed on each channel to obtain a C-dimensional vector. Then, the pooled feature vector is processed by a one-dimensional convolution operation Conv1D(·) to generate a channel vector. Then, the convolution result is normalized using the Sigmoid activation function σ(·) to obtain the final channel weight. Finally, the weight is applied channel by channel to the original input feature map X to obtain the weighted feature map X′, thereby selectively enhancing the features of important channels and suppressing unimportant channel information. Where, X ijc Represents X ijc The value at position (i, j) and channel c.

7. The method for identifying key elements of an operating environment according to claim 6, characterized in that: The S34 includes: using the lightweight feature extraction mechanism β module to replace the standard convolution operation of the decoder part of the original DeepLabV3+ model; in the feature extraction process, the calculation amount of the β module is 1 / K of the standard convolution calculation amount 2 , where K is the size of the convolution kernel; the calculation formulas are as follows: Calculation Amount β =C in ×K 2 ×H×W+C in ×C out ×H×W Calculation Amount 标准卷积 =C in ×C out ×K 2 ×H×W Calculation Amount β / Calculation 标准卷积 =1 / K 2 +1 / C out ≈1 / K 2 .

8. The method for identifying key elements of an operating environment according to claim 7, characterized in that: The S35 includes: using the DF loss function as the loss function of P-Net to deal with the imbalance problem between the target information pixels and the background pixels; First, by optimizing the overlap between the target area and the prediction area, the segmentation accuracy of the target area is enhanced; secondly, by weighting the contribution of the background area, the interference of background pixels on the training process is reduced, making the model more focused on the learning of the target area; the DF calculation formula is as follows: In the formula, ∈ is a small constant, N is the total number of samples, and are the true label and prediction result of the i-th sample respectively.

9. The method for identifying key elements of an operating environment according to claim 3, wherein: The S4 includes: S41, configure training hyperparameters, input the image resolution of the training set as d1*d2, the batch size as B, and the number of iterations as E rounds; S42, start the training model, calculate the model loss value and accuracy value in each training iteration, and update the model weight; S43. During the training process, the P-Net model regularly outputs training indicators, including loss values ​​and accuracy indicators, and verifies them on the validation set to obtain the performance indicators of the model.

10. The method for identifying key elements of an operating environment according to claim 1, characterized in that: The test evaluation described in S5 includes the following indicators: The pixel accuracy PA is calculated as follows: P ii is the total number of pixels belonging to class i and predicted as class i, P ij is the total number of pixels belonging to class i and predicted to be class j, and K+1 is the number of classes; The average pixel accuracy MPA is calculated as follows: P ii is the total number of pixels belonging to class i and predicted as class i, P ij is the total number of pixels belonging to class i and predicted to be class j, and K+1 is the number of classes; The average intersection-over-union ratio MIOU is calculated as follows: P ii is the total number of pixels belonging to class i and predicted as class i, P ji is the total number of pixels that belong to class j but are predicted to be class i, and K+1 is the number of classes. The model optimization in S5 includes: First, based on the evaluation results, analyze the performance of the model in each category, especially focusing on areas with low target detection and segmentation accuracy; second, dynamically adjust the weights in the loss function to enhance the optimization effect on difficult samples and small targets; then, prevent overfitting and improve the generalization ability of the model by adopting a learning rate scheduler and early stopping strategy; finally, fine-tune the structure or hyperparameters of the P-Net model based on the evaluation results to ensure the optimal accuracy of key element recognition in the operating environment and improve the practical application performance and stability of the model.

Images