Method and device for detecting obstacles in unstructured farmland

Abstract

One or more embodiments of the invention disclose a method and a device for detecting obstacles in an unstructured farmland, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target image of an unstructured farmland where an unmanned agricultural machine is located based on at least one camera deployed on the unmanned agricultural machine; inputting the target image into an extraction network of an obstacle detection model, performing multi-scale feature map extraction on the target image, and respectively extracting a channel attention map and a space attention map from the extracted feature map; using a fusion network of the obstacle detection model to perform a first fusion operation on the channel attention map and the space attention map corresponding to the same feature map to obtain a fused feature map, and performing a second fusion operation on the fused feature maps or feature maps of different scales to obtain output features; and inputting the output features into a detection network of the obstacle detection model to obtain an obstacle recognition result corresponding to the target image.

Description

Technical Field

[0001] The present specification relates to the field of data processing technology, and in particular to a method and device, electronic device and storage medium for detecting obstacles in unstructured farmland. Background Art

[0002] Unmanned agricultural machinery (also known as unmanned agricultural machinery) does not require human intervention in operation. It can autonomously perceive the surrounding environment and road information, automatically plan a reasonable driving path, and finally control the agricultural machinery to move along the planned path through the execution control system. Unmanned agricultural machinery can reduce the driver's work intensity and improve work accuracy and production efficiency.

[0003] However, there are no lanes or traffic lights in the farmland environment where unmanned agricultural machinery operates, and the growth of crops changes periodically with the seasons, which increases the difficulty of accurate environmental perception. Unmanned agricultural machinery needs to avoid obstacles based on the typical characteristics of the farmland environment to find a passable path. Therefore, there is an urgent need to provide a better solution for farmland obstacle detection. Summary of the invention

[0004] The embodiments of this specification provide a method and device for detecting obstacles in unstructured farmland, an electronic device, and a storage medium, so as to provide a better solution for detecting obstacles in unstructured farmland.

[0005] In a first aspect, one or more embodiments of this specification provide a method for detecting obstacles in an unstructured farmland, including: Based on at least one camera deployed on the unmanned agricultural machine, a target image of the unstructured farmland where the unmanned agricultural machine is located is acquired; Inputting the target image into an extraction network of an obstacle detection model, extracting a multi-scale feature map of the target image, and respectively extracting a channel attention map and a spatial attention map from the extracted feature map; Using the fusion network of the obstacle detection model, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and a second fusion operation is performed on the fused feature maps or feature maps of different scales to obtain output features; The output features are input into the detection network of the obstacle detection model to obtain an obstacle recognition result corresponding to the target image.

[0006] Optionally, extracting a channel attention map and a spatial attention map from the extracted feature maps respectively includes: From the multi-scale feature maps, obtain the target feature map whose receptive field is smaller than the preset value; For the target feature map, a channel attention map and a spatial attention map are extracted respectively.

[0007] Optionally, before using the extraction network and the fusion network, the method includes: Obtaining a target layer in the extraction network, and using spatial pyramid decomposition convolution in the target layer in the extraction network; and / or A target layer in the fusion network is obtained, and a spatial pyramid decomposition convolution is used in the target layer in the fusion network.

[0008] Optionally, the training process of the obstacle detection model includes: Acquire a training image of the unstructured farmland, and mark obstacles in the training image to obtain a marking result corresponding to the training image; Inputting the training image into the extraction network, extracting a multi-scale training feature map of the training image, and respectively extracting a channel attention map and a spatial attention map from the extracted training feature map; Using the fusion network, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same training feature map to obtain a fused training feature map, and a second fusion operation is performed on the fused training feature maps or training feature maps of different scales to obtain output training features; Inputting the output training features into the detection network to obtain an obstacle recognition result corresponding to the training image; Based on the obstacle recognition result and the labeling result corresponding to the training image, the obstacle detection model is trained to obtain a trained obstacle detection model.

[0009] Optionally, the acquiring the training image of the unstructured farmland includes: Acquire geographic location information of the unstructured farmland, and determine common obstacles in the unstructured farmland based on the geographic location information; According to the common obstacles, a training image of the unstructured farmland is acquired.

[0010] Optionally, the marking obstacles on the training image to obtain a marking result corresponding to the training image includes: Determining the proportion of the object to be labeled in the training image; When the ratio is greater than a preset value, determining a close-up annotation result corresponding to the object to be annotated, wherein the close-up annotation result includes a core area of ​​the object to be annotated; When the ratio is less than a preset value, a distant annotation result corresponding to the object to be annotated is determined, and the distant annotation result includes the entire area of ​​the object to be annotated.

[0011] Optionally, the method further includes: According to the obstacle recognition result corresponding to the target image, determining the avoidance strategy of the unmanned agricultural machinery from a preset strategy library; Based on the avoidance strategy, the path of the unmanned agricultural machine is replanned.

[0012] In a second aspect, an embodiment of the present specification provides a device for detecting obstacles in unstructured farmland, including: An image acquisition module, used for acquiring a target image of an unstructured farmland where the unmanned agricultural machine is located based on at least one camera deployed on the unmanned agricultural machine; A feature extraction module, used to input the target image into an extraction network of an obstacle detection model, extract a multi-scale feature map of the target image, and respectively extract a channel attention map and a spatial attention map from the extracted feature map; A feature fusion module is used to use the fusion network of the obstacle detection model to perform a first fusion operation on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and to perform a second fusion operation on the fused feature maps or feature maps of different scales to obtain output features; The obstacle detection module is used to input the output features into the detection network of the obstacle detection model to obtain the obstacle recognition result corresponding to the target image.

[0013] Optionally, the feature extraction module includes: A target feature map unit, used to obtain a target feature map whose receptive field is smaller than a preset value from a multi-scale feature map; The feature extraction unit is used to extract a channel attention map and a spatial attention map from the target feature map.

[0014] Optionally, the device for detecting obstacles in unstructured farmland comprises: A first convolution module, used for obtaining a target layer in the extraction network before using the extraction network and the fusion network, and using spatial pyramid decomposition convolution in the target layer in the extraction network; and / or The second convolution module is used to obtain a target layer in the fusion network before using the extraction network and the fusion network, and use spatial pyramid decomposition convolution in the target layer in the fusion network.

[0015] Optionally, the training process of the obstacle detection model includes: Acquire a training image of the unstructured farmland, and mark obstacles in the training image to obtain a marking result corresponding to the training image; Inputting the training image into the extraction network, extracting a multi-scale training feature map of the training image, and respectively extracting a channel attention map and a spatial attention map from the extracted training feature map; Using the fusion network, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same training feature map to obtain a fused training feature map, and a second fusion operation is performed on the fused training feature maps or training feature maps of different scales to obtain output training features; Inputting the output training features into the detection network to obtain an obstacle recognition result corresponding to the training image; Based on the obstacle recognition result and the labeling result corresponding to the training image, the obstacle detection model is trained to obtain a trained obstacle detection model.

[0016] Optionally, the acquiring the training image of the unstructured farmland includes: Acquire geographic location information of the unstructured farmland, and determine common obstacles in the unstructured farmland based on the geographic location information; According to the common obstacles, a training image of the unstructured farmland is acquired.

[0017] Optionally, the marking obstacles on the training image to obtain a marking result corresponding to the training image includes: Determining the proportion of the object to be labeled in the training image; When the ratio is greater than a preset value, determining a close-up annotation result corresponding to the object to be annotated, wherein the close-up annotation result includes a core area of ​​the object to be annotated; When the ratio is less than a preset value, a distant annotation result corresponding to the object to be annotated is determined, and the distant annotation result includes the entire area of ​​the object to be annotated.

[0018] Optionally, the feature extraction module further includes: An avoidance strategy module, used to determine the avoidance strategy of the unmanned agricultural machinery from a preset strategy library according to the obstacle recognition result corresponding to the target image; A path planning module is used to replan the path of the unmanned agricultural machine based on the avoidance strategy.

[0019] In a third aspect, an embodiment of the present specification provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present specification provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0021] In a fifth aspect, an embodiment of this specification provides a chip, the chip comprising a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0022] In a sixth aspect, an embodiment of this specification provides a computer program product, which, when executed by a processor, implements the method described in the first aspect.

[0023] In the embodiment of this specification, after extracting multi-scale feature maps from the target image through the extraction network, the channel attention map and the spatial attention map are extracted from the feature map, and then the channel attention map and the spatial attention map of the same scale are fused using the first fusion operation through the fusion network, and the fused feature maps or feature maps of different scales are fused using the second fusion operation, and finally the obstacle detection result corresponding to the fused input features is determined through the detection network. This process can improve the detection accuracy of obstacles (especially small target obstacles) of the obstacle detection model by extracting channel attention maps and spatial attention maps of feature maps of multiple scales, and realize accurate perception of the environment in which the unmanned agricultural machinery is located, thereby improving the obstacle avoidance effect of the unmanned agricultural machinery when autonomously avoiding obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flowchart of a method for detecting obstacles in unstructured farmland according to an embodiment of the present specification.

[0026] Figure 2 It is a schematic diagram of a framework of an obstacle detection model constructed based on the YOLOv8 algorithm according to an embodiment of this specification.

[0027] Figure 3 It is a structural diagram of an SA module according to an embodiment of this specification.

[0028] Figure 4This is a module detail diagram of an obstacle detection model based on an improved YOLOv8 algorithm according to an embodiment of this specification.

[0029] Figure 5 It is a flowchart of a training process of an obstacle detection model according to an embodiment of the present specification.

[0030] Figure 6 This is an example diagram of obstacles in a data set according to an embodiment of the present specification.

[0031] Figure 7 It is a schematic diagram of the working process of a system for detecting obstacles in unstructured farmland according to one embodiment of the present specification.

[0032] Figure 8 It is a structural schematic diagram of a device for detecting obstacles in unstructured farmland according to an embodiment of the present specification.

[0033] Fig. 9 It is a structural schematic diagram of an electronic device according to an embodiment of this specification. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of this specification can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] Figure 1A flow chart of a method for detecting obstacles in unstructured farmland according to an embodiment of the present disclosure is shown, and the method can be applied to an apparatus for detecting obstacles in unstructured farmland, and the apparatus for detecting obstacles in unstructured farmland can be a terminal device, a server or other processing device, etc. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.

[0037] In some possible implementations, the method for detecting obstacles in unstructured farmland may be implemented by a processor calling computer-readable instructions stored in a memory.

[0038] like Figure 1 As shown, the method for detecting obstacles in unstructured farmland may include: Step S102: Based on at least one camera deployed on the unmanned agricultural machine, a target image of the unstructured farmland where the unmanned agricultural machine is located is acquired.

[0039] Among them, unmanned agricultural machinery is unmanned agricultural machinery, and the unmanned agricultural machinery can be a seeding machine, a harvester, etc. The type of unmanned agricultural machinery is not specifically limited in this specification.

[0040] Unstructured farmland refers to farmland without fixed structures or design features, usually those that are naturally formed without human intervention or transformation. This type of farmland is characterized by complex terrain, diverse soil types, random crop planting, and lack of unified management and planning. Obstacles in unstructured farmland vary in shape and size. In addition to static targets such as telephone poles and farm tools that affect safe passage, obstacles in unstructured farmland also include dynamic targets such as people, animals, and other moving farm machinery, which brings great difficulties to the obstacle recognition task.

[0041] Environmental perception is the primary condition for the intelligentization of agricultural equipment. In order to realize the environmental perception of unmanned agricultural machinery, cameras can be deployed on the unmanned agricultural machinery to collect the target image of the farmland where the unmanned agricultural machinery is located. In one example, ROS2.0 can be used to connect the above camera to the device, and the ros2 topic list can be used to check whether the connection is successful. After the connection is successful, the obstacle can be photographed. After the shooting is completed, the command ros2 launch ros2_bag_to_image bag_to_image.xml can be used to export the image data in the .bag file. Specifically, the camera parameters can be shown in Table 1: Table 1 Camera information table

[0042] The number of cameras can be one or more. When there is only one camera, the camera can be deployed on the top of the unmanned agricultural machine; when there are multiple cameras, the cameras can be deployed on the top, side, rear, etc. of the unmanned agricultural machine to achieve better perception of the environment in front, side, and even rear of the unmanned agricultural machine.

[0043] Step S104: input the target image into the extraction network of the obstacle detection model, extract multi-scale feature maps of the target image, and respectively extract channel attention maps and spatial attention maps from the extracted feature maps.

[0044] The obstacle detection model is a detection model deployed on the unmanned agricultural machinery, which detects obstacles in the surrounding environment of the unmanned agricultural machinery through the target image collected by the camera. The obstacle detection model can include an extraction network for extracting image features, a fusion network for fusing the extracted image features, and a detection network for obstacle detection based on the fused features.

[0045] In one example, the obstacle detection model may be a model built based on the YOLOv8 algorithm. Figure 2 The framework of the obstacle detection model built based on the YOLOv8 algorithm is shown. The obstacle detection model includes a backbone network (Backbone), a neck network (Neck) and a head network (Head). The backbone network (Backbone) can be used as an extraction network, the neck network (Neck) can be used as a fusion network, and the head network (Head) can be used as a detection network.

[0046] Since the deep network has a large receptive field and strong ability to represent semantic information, but the resolution of the feature map is low and the ability to represent geometric information is weak (lack of spatial geometric feature details), while the low-level network has a small receptive field and strong ability to represent geometric detail information. Although the resolution is high, the ability to represent semantic information is weak. Therefore, the extraction network can be set to a network based on multiple convolution kernels. In this way, the obstacle features can be extracted by inputting the target image into the extraction network, and feature maps of various scales can be obtained, so that the feature extraction process can take into account both the global features and local detail features of the target image.

[0047] The attention mechanism enables a neural network to accurately focus on all relevant elements of the input, and it has become an important component for improving the performance of deep neural networks. In one example, an attention module can be set in the extraction network, which can extract a spatial attention map and a channel attention map from a feature map, both of which capture the pixel-level pairwise relationship and channel dependency in the feature map, respectively. Since capturing both the spatial attention map and the channel attention map from the feature map increases the computational overhead, the Shuffle Attention (SA) module can be used as an attention module to effectively combine the two attention mechanisms while maintaining efficient computation.

[0048] Specifically, the SA module combines the two types of attention mechanisms, spatial attention and channel attention. It first splits the input into multiple groups along the channel dimension, then uses a permutation unit to characterize the dependencies of each group of features in the spatial and channel dimensions, and finally integrates all the features and performs component feature communication through channel permutation operations. Figure 3 FIG. 2 shows a schematic diagram of the structure of the SA module. Figure 3 As shown, the SA module can include the following four parts: a) Feature Grouping: Mainly used to group input features. The input feature is X∈R H×W×C , split X into G groups along the channel dimension: X=[X1,…X G ],R C / G×H×W Then the feature X K Split into two branches: X K1 and X K2 , one branch is used to learn channel attention features, one is used to learn spatial attention features, and X K1 ∈R C / 2G×H×W , X K2 ∈R C / 2G×H×W .

[0049] b) Channel Attention: In terms of channel attention implementation, this module uses the simplest combination of GAP+Scale+Sigmoid. The specific formula description is shown in Formula 1 and Formula 2: (Formula 1) (Formula 2) in, represents the channel attention map, X k1 Represents the input feature map, i∈H, j∈W, H, W represent the height and width of the input feature map, F gp(.) represents the global average pooling operation, σ represents the Sigmoid function, and F c (.) represents the fully connected operation, s represents the pooling result, and the above two formulas have only two changing parameters, namely W1∈R C / 2G×1×1 , b1∈R C / 2G×1×1 , G represents the feature map X k1 The number of splits, C represents the number of channels.

[0050] c) Spatial Attention: First, Group Norm (GN) is used to K2 Processing is performed to obtain statistical information at the spatial level, and then the F c (.) can be enhanced. Formula 3 can be used to achieve X K2 To process: (Formula 3) in, represents the spatial attention map, GN(.) represents the normalization operation, X k2 Represents the input feature map, σ represents the Sigmoid function, and there are two changing parameters in this formula, namely W2∈R C / 2G×1×1 , b2∈R C / 2G×1×1 , G represents the feature map X k2 The number of splits, C represents the number of channels.

[0051] d) Aggregation: After completing the previous two attention calculations, integrate them and fuse them through a simple Concat operation to obtain the fusion feature , at this time, the size of the fused feature is consistent with the input size.

[0052] The extraction of multi-scale feature maps can take into account both the global features and local detail features of the target image, but setting an attention module for each network that extracts feature maps will undoubtedly increase the computational overhead. In one implementation, the extraction of channel attention maps and spatial attention maps from the extracted feature maps includes: From the multi-scale feature maps, obtain the target feature map whose receptive field is smaller than the preset value; For the target feature map, a channel attention map and a spatial attention map are extracted respectively.

[0053] Specifically, attention modules can be set only for low-level networks that process small targets. For different networks in the extraction network, the low-level network and high-level network can be determined based on the size of the receptive field of the extracted feature map, and attention modules can be configured for the determined low-level networks. The feature maps extracted by these low-level networks are the target feature maps. In this way, after obtaining multi-scale feature maps, channel attention maps and spatial attention maps can be extracted from the target feature maps respectively, so as to reduce the computational overhead while ensuring the accuracy of the feature maps.

[0054] Figure 4 The module details of the obstacle detection model based on the improved YOLOv8 are shown. Figure 4 As shown in the figure, in the backbone network of the third and fourth layers as the low-level network, the SA attention module is combined with the C2f module to construct the C2f-Att module. The C2f-Att module extracts the channel attention map and the spatial attention map, so that the obstacle detection model enhances the feature extraction capability of small target obstacles in the image containing obstacles, thereby enabling the obstacle detection model to obtain better detection accuracy in the real farmland environment (including uneven lighting and high noise).

[0055] Step S106: Using the fusion network of the obstacle detection model, the channel attention map and the spatial attention map corresponding to the same feature map are subjected to a first fusion operation to obtain a fused feature map, and the fused feature map or feature maps of different scales are subjected to a second fusion operation to obtain an output feature.

[0056] After obtaining the channel attention map and the spatial attention map, the fusion network can be used for fusion operation. Specifically, the first fusion operation can be performed on the channel attention map and the spatial attention map of the same feature map to obtain a fused feature map, and then the second fusion operation can be performed on the feature maps and / or fused feature maps of different scales to obtain the output features corresponding to the fusion network.

[0057] In one example, the process of the second fusion operation may include: when there is a corresponding fusion feature map at the target scale (referring to any scale), using the fusion feature map corresponding to the target scale as the feature map to be fused; when there is no corresponding fusion feature map at the target scale, using the feature map corresponding to the target scale as the feature map to be fused; performing a second fusion operation on the feature maps to be fused at each scale to obtain the output feature.

[0058] Step S108: inputting the output features into the detection network of the obstacle detection model to obtain an obstacle recognition result corresponding to the target image.

[0059] Specifically, the result of multi-scale feature map fusion (i.e., output feature) can be input into the detection network to determine whether there is an obstacle in the target image and the type of the obstacle.

[0060] In the embodiment of this specification, after extracting multi-scale feature maps from the target image through the extraction network, the channel attention map and the spatial attention map are extracted from the feature map, and then the channel attention map and the spatial attention map of the same scale are fused using the first fusion operation through the fusion network, and the fused feature maps or feature maps of different scales are fused using the second fusion operation, and finally the obstacle detection result corresponding to the fused input features is determined through the detection network. This process can improve the detection accuracy of obstacles (especially small target obstacles) of the obstacle detection model by extracting channel attention maps and spatial attention maps of feature maps of multiple scales, and realize accurate perception of the environment in which the unmanned agricultural machinery is located, thereby improving the obstacle avoidance effect of the unmanned agricultural machinery when autonomously avoiding obstacles.

[0061] During the operation of unmanned agricultural machinery, the dust generated may cause the image captured by the camera to be blurred, have low resolution, and other problems. In one implementation, before using the extraction network and the fusion network, the method includes: Obtaining a target layer in the extraction network, and using spatial pyramid decomposition convolution in the target layer in the extraction network; and / or A target layer in the fusion network is obtained, and a spatial pyramid decomposition convolution is used in the target layer in the fusion network.

[0062] Fine-grained information is crucial to the accuracy of obstacle detection results. In addition to extracting channel attention maps and spatial attention maps, other methods can be used to better collect fine-grained information of the target image.

[0063] Traditional convolutional neural networks (CNNs) often lose fine-grained information when processing low-resolution images and small objects, resulting in poor detection results. Space-to-Depth Convolution (SPD) can retain all information in the channel dimension during the downsampling process, so that when processing low-resolution images, it can better capture detailed information, thereby improving detection accuracy and efficiency.

[0064] The target layer is a network layer that uses spatial pyramid decomposition convolution. Specifically, spatial pyramid decomposition convolution can be used for the target layer in the extraction network and / or the fusion network. When the target layer is a network layer in the extraction network, using spatial pyramid decomposition convolution can retain more fine-grained information in the feature extraction process; when the target layer is a network layer in the fusion network, using spatial pyramid decomposition convolution can retain more fine-grained information in the multi-scale feature fusion process.

[0065] In one example, the target layer in the extraction network can be determined based on the size of the receptive field corresponding to each feature map. For example, in the extraction network, the network layer where the feature map with a larger receptive field (for example, the receptive field is larger than a preset value) is located can be used as the target layer to improve the extraction effect of the network layer on the obstacle features in the low-resolution image.

[0066] like Figure 4 As shown in the figure, in the backbone network of the second layer, Conv-SPD can be used to replace the original block to retain more fine-grained information; Conv-SPD can be added to the neck network to retain more fine-grained information in the fusion process of multi-scale features. Specifically, after the original block is replaced by Conv-SPD in the backbone network of the second layer (layer2), the specific operations of layer2 include: (a) Get the feature map X corresponding to layer2, the size of X is S×S×C1; (b) Downsample using the scale factor (scale), dividing X into scale 2 Sub-feature map f x,y Each sub-feature map is a part of X, where the index (i) and (j) in X(i,j) are divisible by scale. By reducing the spatial resolution of the feature map, the depth (number of channels) of the feature map can be increased; (c) These sub-features are merged along the channel dimension to form a new feature map X0. The spatial dimension of X0 becomes 1 / scale of the original, but the number of channels increases by scale 2 Then a convolution layer with a stride of 1 is applied to X0, which means that every pixel of the feature map will be covered by the convolution kernel and no information will be skipped.

[0067] Through this method, the original feature map X undergoes a series of transformations, the spatial resolution is reduced, but the number of channels is increased, and a more discriminative feature representation can be obtained after non-compensatory convolution.

[0068] In the above process, by using spatial pyramid decomposition convolution in the extraction network and / or the fusion network, the depth of the feature map can be increased without losing information, thereby avoiding information loss in traditional stride convolution and pooling operations, and retaining the fine-grained information of the extraction network and / or the fusion network during the operation process, thereby improving the detection effect of obstacles in low-resolution target images.

[0069] In one implementation, the training process of the obstacle detection model includes: Acquire a training image of the unstructured farmland, and mark obstacles in the training image to obtain a marking result corresponding to the training image; Inputting the training image into the extraction network, extracting a multi-scale training feature map of the training image, and respectively extracting a channel attention map and a spatial attention map from the extracted training feature map; Using the fusion network, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same training feature map to obtain a fused training feature map, and a second fusion operation is performed on the fused training feature maps or training feature maps of different scales to obtain output training features; Inputting the output training features into the detection network to obtain an obstacle recognition result corresponding to the training image; Based on the obstacle recognition result and the labeling result corresponding to the training image, the obstacle detection model is trained to obtain a trained obstacle detection model.

[0070] Figure 5 FIG. 1 is a flow chart showing the training process of the obstacle detection model. Figure 5 As shown, the training process of the obstacle detection model is similar to the aforementioned detection process, and may include steps such as training image acquisition, feature map extraction, feature fusion, and obstacle detection.

[0071] In the training image acquisition step, the camera deployed on the unmanned agricultural machinery can be used to shoot farmland obstacle videos in different time periods, seasons, and weather environments during the operation of the unmanned agricultural machinery. The captured videos can be framed to construct an original data set. Next, the collected images containing obstacles can be preprocessed to construct a standard data set for training the farmland obstacle detection model, so as to train the obstacle detection model with the data in the standard data set. In addition, due to the different heights of different agricultural machinery, the angles of obstacles captured by the camera are different. When making the training set, the shooting angle of the training image can be considered to make the shooting angle of the training image as consistent as possible with the shooting angle of the aforementioned target image, so as to ensure the accuracy of the obstacle detection method in unstructured farmland.

[0072] In one example, a camera can be connected to the Ros system and fixed above an unmanned agricultural machine. During the operation of the agricultural machine, the camera can be used to record obstacles. After the shooting is completed, the original data set can be constructed by extracting the image file from the recorded .bag file.

[0073] The types of obstacles in the surrounding environment of the unmanned agricultural machinery vary depending on the geographical location of the unstructured farmland. For example, in some farmlands in the north, obstacles are usually sheep, people working on the farm, etc., while in some farmlands in the south, obstacles are usually trees, electric poles, agricultural machinery, people working on the farm, etc. In one implementation, obtaining the training image of the unstructured farmland includes: Acquire geographic location information of the unstructured farmland, and determine common obstacles in the unstructured farmland based on the geographic location information; According to the common obstacles, a training image of the unstructured farmland is acquired.

[0074] Specifically, based on the geographic location information of the unstructured farmland, common obstacles in the unstructured farmland can be determined, and a camera can be used to obtain training images containing these common obstacles. This can focus on improving the detection accuracy of the obstacle detection model for these common obstacles.

[0075] Furthermore, image annotation tools (such as Labelimg, etc.) can be used to annotate the above-extracted original data set. Since the proportion of the same obstacle in the image is different when the camera captures a close-up or long-range image, the obstacle annotation of the training image to obtain the annotation result corresponding to the training image includes: Determining the proportion of the object to be labeled in the training image; When the ratio is greater than a preset value, determining a close-up annotation result corresponding to the object to be annotated; When the ratio is smaller than a preset value, a distant view annotation result corresponding to the object to be annotated is determined.

[0076] The object to be labeled is an obstacle in the training image. The near-view labeling result includes the core area of ​​the object to be labeled, and the far-view labeling result includes the entire area of ​​the object to be labeled. For example, when the object to be labeled is a tree, for the older trees in the near-view, the core feature areas such as the trunk and branches are labeled, and for the short landscape trees in the distance, the entire area including the leaves is labeled.

[0077] In one example, the near-view image and the distant view image can be distinguished by information such as the proportion of the object to be annotated in the training image. At the same time, the preset values ​​corresponding to each annotated object should be different. It should be understood that the preset values ​​here are different from the preset values ​​used for extracting the channel attention map and the spatial attention map mentioned above, and they all need to be determined according to the actual situation. This manual does not make a specific distinction in the name.

[0078] In the above process, the objects to be labeled are labeled according to their proportion in the training image, thereby achieving the distinction between close-up and distant annotations of the objects to be labeled. This helps to improve the detection effect of the obstacle detection model for obstacles of different forms and improve the robustness of the obstacle detection model.

[0079] After the training images are labeled, the labeled data set can also be preprocessed by resizing, normalizing, and other operations. When there is less data in the data set, the constructed standard data set can also be augmented online to expand the data set and improve the diversity and generalization of the model data set. In one example, the data augmentation operation can include: mosaic enhancement, mixup, random perspective, and color perturbation (HSV augment).

[0080] Next, the preprocessed data set can be randomly divided into a training set and a test set according to a certain ratio (such as 9:1) to train and test the obstacle detection model respectively. The number of various obstacles in the data set can be shown in Table 2, and the example diagram of obstacles in the data set can be shown in Figure 6 shown.

[0081] Table 2 Overall situation of the data set

[0082] In the feature map extraction step, the training set can be normalized and input into the extraction network built based on the attention mechanism to extract obstacle features, and obtain feature maps of various scales. Furthermore, channel attention maps and spatial attention maps can be extracted from the training feature maps to improve the recognition accuracy and precision of the obstacle detection model for small targets.

[0083] In the feature fusion step, the channel attention map and the spatial attention map of the same scale can be first fused through a fusion network to obtain a fused training feature map, and the fused training feature map or the training feature map of different scales can be secondly fused to obtain output training features.

[0084] In the obstacle detection step, the output training features can be input into the detection network to obtain the obstacle recognition result corresponding to the training image. In this way, a loss function can be constructed based on the gap between the obstacle recognition result and the annotation result to optimize the parameters in the model. When the loss corresponding to the loss function is less than or equal to the set value or the number of training times reaches the set number (such as 50 times), the training ends and a trained obstacle detection model is obtained.

[0085] In order to comprehensively reflect the performance of the model, the evaluation indicators selected for the obstacle detection model may include precision (P), recall (R) and mean of average precision (P mA ). P represents the ratio of the predicted algorithm area to the actual detection area, R represents the ratio of the accurately predicted categories to the total number of required categories, P mA Based on the precision-recall curve, the accuracy and robustness of the algorithm can be comprehensively evaluated. mA The higher the value, the higher the prediction accuracy. The specific formula of the evaluation index is shown in Formula 4.

[0086] (Formula 4) Among them, T P represents true positive, F P represents false positive, N represents the total number of samples, F N represents the number of true positives that are missed, P iA It represents the average value of the average accuracy of each category of the total number of obstacle categories detected, P mA represents the mean average precision, P represents the accuracy, and R represents the recall.

[0087] Furthermore, inference time and network parameter volume can also be used as performance indicators. Completing inference in a shorter time means that the model has better real-time performance, while the model size represents memory usage.

[0088] Table 3 shows the performance of the obstacle detection model obtained by training data with the geographical location of unstructured farmland in the south. As shown in Table 3, the obstacle detection model is P mA 50 is 97.01%, which means that the obstacle detection model built based on the improved YOLOv8 algorithm has better detection effect than the obstacle detection model built based on the original YOLOv8 algorithm.

[0089] Table 3 Performance of obstacle detection model

[0090] In one implementation, the method further includes: According to the obstacle recognition result corresponding to the target image, determining the avoidance strategy of the unmanned agricultural machinery from a preset strategy library; Based on the avoidance strategy, the driving path of the unmanned agricultural machinery is replanned.

[0091] Specifically, for different obstacles, the avoidance strategies of the unmanned agricultural machinery (including whether to avoid, the avoidance distance, etc.) are different. For example, for obstacles such as people or agricultural machinery, the avoidance strategy may include honking the horn, signaling people or agricultural machinery to avoid, etc., while for obstacles such as trees and electric poles, the avoidance strategy may include setting the area within the target distance of the obstacle as the avoidance area, etc.

[0092] In one example, experts in the field of agricultural machinery can discuss and formulate avoidance strategies for different obstacles, and save the determined avoidance strategies in a preset strategy library. In this way, after the unmanned agricultural machinery determines the obstacle recognition result of the target image, it can determine the avoidance strategy of the unmanned agricultural machinery that matches the obstacle recognition result from the strategy library. Furthermore, after determining the avoidance strategy, the initial driving path of the unmanned agricultural machinery can be changed to obtain a new driving path.

[0093] In one example, a system for detecting obstacles in unstructured farmland may be constructed. The system may include: at least one processor, a camera, a memory, and instructions for detecting obstacles in unstructured farmland. Figure 7 The schematic diagram of the working process of the system for obstacle detection in unstructured farmland is shown in FIG. Figure 7 As shown in the figure, the system installs a camera above the unmanned agricultural machinery so that the unmanned agricultural machinery can perceive the surrounding environment, extracts images from the video data at 3 frames per second and inputs them into the obstacle detection model. The obstacle detection model is used to perform obstacle detection on the collected images to obtain obstacle detection results. The unmanned agricultural machinery performs autonomous obstacle avoidance based on the obstacle detection results.

[0094] In the above process, based on the obstacle recognition results determined by the obstacle detection model, the driving path of the unmanned agricultural machinery is replanned, the dynamic adjustment of the working path of the unmanned agricultural machinery is realized, and the safety of the operation process of the unmanned agricultural machinery is improved.

[0095] It can be understood that the above-mentioned various method embodiments mentioned in this specification can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, they will not be repeated in this specification. Those skilled in the art can understand that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0096] It should be noted that the method for detecting obstacles in unstructured farmland provided in the embodiments of this specification may be performed by a device for detecting obstacles in unstructured farmland, or a control module in the device for detecting obstacles in unstructured farmland for performing the method for detecting obstacles in unstructured farmland. In the embodiments of this specification, the method for detecting obstacles in unstructured farmland performed by the device for detecting obstacles in unstructured farmland is taken as an example to illustrate the device for detecting obstacles in unstructured farmland provided in the embodiments of this specification.

[0097] Figure 8 FIG. 1 is a schematic diagram of the structure of an apparatus for detecting obstacles in unstructured farmland according to an embodiment of the present invention. Figure 8 As shown, the device 800 for detecting obstacles in unstructured farmland includes: An image acquisition module 810 is used to acquire a target image of an unstructured farmland where the unmanned agricultural machine is located based on at least one camera deployed on the unmanned agricultural machine; A feature extraction module 820 is used to input the target image into an extraction network of an obstacle detection model, extract a multi-scale feature map of the target image, and respectively extract a channel attention map and a spatial attention map from the extracted feature map; A feature fusion module 830 is used to use the fusion network of the obstacle detection model to perform a first fusion operation on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and to perform a second fusion operation on the fused feature maps or feature maps of different scales to obtain an output feature; The obstacle detection module 840 is used to input the output features into the detection network of the obstacle detection model to obtain the obstacle recognition result corresponding to the target image.

[0098] In one embodiment, the feature extraction module 820 includes: A target feature map unit, used to obtain a target feature map whose receptive field is smaller than a preset value from a multi-scale feature map; The feature extraction unit is used to extract a channel attention map and a spatial attention map from the target feature map.

[0099] In one embodiment, the device 800 for detecting obstacles in unstructured farmland includes: A first convolution module, used for obtaining a target layer in the extraction network before using the extraction network and the fusion network, and using spatial pyramid decomposition convolution in the target layer in the extraction network; and / or The second convolution module is used to obtain a target layer in the fusion network before using the extraction network and the fusion network, and use spatial pyramid decomposition convolution in the target layer in the fusion network.

[0100] In one embodiment, the training process of the obstacle detection model includes: Acquire a training image of the unstructured farmland, and mark obstacles in the training image to obtain a marking result corresponding to the training image; Inputting the training image into the extraction network, extracting a multi-scale training feature map of the training image, and respectively extracting a channel attention map and a spatial attention map from the extracted training feature map; Using the fusion network, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same training feature map to obtain a fused training feature map, and a second fusion operation is performed on the fused training feature maps or training feature maps of different scales to obtain output training features; Inputting the output training features into the detection network to obtain an obstacle recognition result corresponding to the training image; Based on the obstacle recognition result and the labeling result corresponding to the training image, the obstacle detection model is trained to obtain a trained obstacle detection model.

[0101] In one embodiment, the acquiring of the training image of the unstructured farmland includes: Acquire geographic location information of the unstructured farmland, and determine common obstacles in the unstructured farmland based on the geographic location information; According to the common obstacles, a training image of the unstructured farmland is acquired.

[0102] In one embodiment, the step of marking obstacles on the training image to obtain a marking result corresponding to the training image includes: Determining the proportion of the object to be labeled in the training image; When the ratio is greater than a preset value, determining a close-up annotation result corresponding to the object to be annotated, wherein the close-up annotation result includes a core area of ​​the object to be annotated; When the ratio is less than a preset value, a distant annotation result corresponding to the object to be annotated is determined, and the distant annotation result includes the entire area of ​​the object to be annotated.

[0103] In one embodiment, the feature extraction module 820 further includes: An avoidance strategy module, used to determine the avoidance strategy of the unmanned agricultural machinery from a preset strategy library according to the obstacle recognition result corresponding to the target image; A path planning module is used to replan the path of the unmanned agricultural machine based on the avoidance strategy.

[0104] In the embodiment of this specification, after extracting multi-scale feature maps from the target image through the extraction network, the channel attention map and the spatial attention map are extracted from the feature map, and then the channel attention map and the spatial attention map of the same scale are fused using the first fusion operation through the fusion network, and the fused feature maps or feature maps of different scales are fused using the second fusion operation, and finally the obstacle detection result corresponding to the fused input features is determined through the detection network. This process can improve the detection accuracy of obstacles (especially small target obstacles) of the obstacle detection model by extracting channel attention maps and spatial attention maps of feature maps of multiple scales, and realize accurate perception of the environment in which the unmanned agricultural machinery is located, thereby improving the obstacle avoidance effect of the unmanned agricultural machinery when autonomously avoiding obstacles.

[0105] Each module in the above-mentioned device for detecting obstacles in unstructured farmland can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the terminal device or the processor on the server side in the form of hardware, or can be stored in the memory in the terminal device or the memory on the server side in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0106] Furthermore, corresponding to the above-described method for detecting obstacles in unstructured farmland, based on the same technical concept, one or more embodiments of the present application further provide an electronic device, such as Fig. 9 As shown. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 901 and memory 902, and the memory 902 may store one or more storage applications or data. Among them, the memory 902 may be a temporary storage or a permanent storage. The application stored in the memory 902 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 901 can be configured to communicate with the memory 902 to execute a series of computer executable instructions in the memory 902 on the electronic device. The electronic device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input and output interfaces 905, and one or more keyboards 906.

[0107] In a specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs configured to be executed by one or more processors include the following computer executable instructions: Based on at least one camera deployed on the unmanned agricultural machine, a target image of the unstructured farmland where the unmanned agricultural machine is located is acquired; Inputting the target image into an extraction network of an obstacle detection model, extracting a multi-scale feature map of the target image, and respectively extracting a channel attention map and a spatial attention map from the extracted feature map; Using the fusion network of the obstacle detection model, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and a second fusion operation is performed on the fused feature maps or feature maps of different scales to obtain output features; The output features are input into the detection network of the obstacle detection model to obtain an obstacle recognition result corresponding to the target image.

[0108] In the embodiment of this specification, after extracting multi-scale feature maps from the target image through the extraction network, the channel attention map and the spatial attention map are extracted from the feature map, and then the channel attention map and the spatial attention map of the same scale are fused using the first fusion operation through the fusion network, and the fused feature maps or feature maps of different scales are fused using the second fusion operation, and finally the obstacle detection result corresponding to the fused input features is determined through the detection network. This process can improve the detection accuracy of obstacles (especially small target obstacles) of the obstacle detection model by extracting channel attention maps and spatial attention maps of feature maps of multiple scales, and realize accurate perception of the environment in which the unmanned agricultural machinery is located, thereby improving the obstacle avoidance effect of the unmanned agricultural machinery when autonomously avoiding obstacles.

[0109] It should be noted that the embodiment of the electronic device in the present application and the embodiment of the method for detecting obstacles in unstructured farmland in the present application are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method for detecting obstacles in unstructured farmland mentioned above, and the repeated parts will not be repeated.

[0110] Further, corresponding to the above-described method for detecting obstacles in unstructured farmland, based on the same technical concept, one or more embodiments of the present application further provide a storage medium for storing computer executable instructions. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instructions stored in the storage medium are executed by the processor, the following process can be implemented: Based on at least one camera deployed on the unmanned agricultural machine, a target image of the unstructured farmland where the unmanned agricultural machine is located is acquired; Inputting the target image into an extraction network of an obstacle detection model, extracting a multi-scale feature map of the target image, and respectively extracting a channel attention map and a spatial attention map from the extracted feature map; Using the fusion network of the obstacle detection model, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and a second fusion operation is performed on the fused feature maps or feature maps of different scales to obtain output features; The output features are input into the detection network of the obstacle detection model to obtain an obstacle recognition result corresponding to the target image.

[0111] In the embodiment of this specification, after extracting multi-scale feature maps from the target image through the extraction network, the channel attention map and the spatial attention map are extracted from the feature map, and then the channel attention map and the spatial attention map of the same scale are fused using the first fusion operation through the fusion network, and the fused feature maps or feature maps of different scales are fused using the second fusion operation, and finally the obstacle detection result corresponding to the fused input features is determined through the detection network. This process can improve the detection accuracy of obstacles (especially small target obstacles) of the obstacle detection model by extracting channel attention maps and spatial attention maps of feature maps of multiple scales, and realize accurate perception of the environment in which the unmanned agricultural machinery is located, thereby improving the obstacle avoidance effect of the unmanned agricultural machinery when autonomously avoiding obstacles.

[0112] It should be noted that the embodiment of the storage medium in the present application and the method for obstacle detection in unstructured farmland in the present application are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method for obstacle detection in unstructured farmland mentioned above, and the repeated parts will not be repeated.

[0113] Furthermore, corresponding to the above-described method for detecting obstacles in unstructured farmland, based on the same technical concept, one or more embodiments of the present application further provide a computer program product, the computer program product including a computer program, which can implement the following process when executed by a processor: Based on at least one camera deployed on the unmanned agricultural machine, a target image of the unstructured farmland where the unmanned agricultural machine is located is acquired; Inputting the target image into an extraction network of an obstacle detection model, extracting a multi-scale feature map of the target image, and respectively extracting a channel attention map and a spatial attention map from the extracted feature map; Using the fusion network of the obstacle detection model, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and a second fusion operation is performed on the fused feature maps or feature maps of different scales to obtain output features; The output features are input into the detection network of the obstacle detection model to obtain an obstacle recognition result corresponding to the target image.

[0114] In the embodiment of this specification, after extracting multi-scale feature maps from the target image through the extraction network, the channel attention map and the spatial attention map are extracted from the feature map, and then the channel attention map and the spatial attention map of the same scale are fused using the first fusion operation through the fusion network, and the fused feature maps or feature maps of different scales are fused using the second fusion operation, and finally the obstacle detection result corresponding to the fused input features is determined through the detection network. This process can improve the detection accuracy of obstacles (especially small target obstacles) of the obstacle detection model by extracting channel attention maps and spatial attention maps of feature maps of multiple scales, and realize accurate perception of the environment in which the unmanned agricultural machinery is located, thereby improving the obstacle avoidance effect of the unmanned agricultural machinery when autonomously avoiding obstacles.

[0115] It should be noted that the embodiment of the computer program product in the present application and the embodiment of the method for detecting obstacles in unstructured farmland in the present application are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method for detecting obstacles in unstructured farmland mentioned above, and the repeated parts will not be repeated.

[0116] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0118] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0119] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0120] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0121] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0126] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0127] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0129] One or more embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present application may also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0130] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0131] The above are only embodiments of this document and are not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A method for obstacle detection in unstructured farmland, characterized in that: include: Based on at least one camera deployed on the unmanned agricultural machine, a target image of the unstructured farmland where the unmanned agricultural machine is located is acquired; Inputting the target image into an extraction network of an obstacle detection model, extracting a multi-scale feature map of the target image, and respectively extracting a channel attention map and a spatial attention map from the extracted feature map; Using the fusion network of the obstacle detection model, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and a second fusion operation is performed on the fused feature maps or feature maps of different scales to obtain output features; The output features are input into the detection network of the obstacle detection model to obtain an obstacle recognition result corresponding to the target image.

2. The method according to claim 1, characterized in that The extracted feature maps are respectively extracted with a channel attention map and a spatial attention map, including: From the multi-scale feature maps, obtain a target feature map whose receptive field is smaller than a preset value; For the target feature map, a channel attention map and a spatial attention map are extracted respectively.

3. The method according to claim 2, characterized in that Before using the extraction network and the fusion network, the method comprises: Obtaining a target layer in the extraction network, and using spatial pyramid decomposition convolution in the target layer in the extraction network; and / or A target layer in the fusion network is obtained, and a spatial pyramid decomposition convolution is used in the target layer in the fusion network.

4. The method according to claim 1, characterized in that: The training process of the obstacle detection model includes: Acquire a training image of the unstructured farmland, and mark obstacles in the training image to obtain a marking result corresponding to the training image; Inputting the training image into the extraction network, extracting a multi-scale training feature map of the training image, and respectively extracting a channel attention map and a spatial attention map from the extracted training feature map; Using the fusion network, a first fusion operation is performed on the channel attention map and the spatial attention map corresponding to the same training feature map to obtain a fused training feature map, and a second fusion operation is performed on the fused training feature maps or training feature maps of different scales to obtain output training features; Inputting the output training features into the detection network to obtain an obstacle recognition result corresponding to the training image; Based on the obstacle recognition result and the labeling result corresponding to the training image, the obstacle detection model is trained to obtain a trained obstacle detection model.

5. The method according to claim 4, characterized in that The obtaining of the training image of the unstructured farmland comprises: Acquire geographic location information of the unstructured farmland, and determine common obstacles in the unstructured farmland based on the geographic location information; According to the common obstacles, a training image of the unstructured farmland is acquired.

6. The method according to claim 5, characterized in that The marking of obstacles on the training image to obtain a marking result corresponding to the training image includes: Determining the proportion of the object to be labeled in the training image; When the ratio is greater than a preset value, determining a close-up annotation result corresponding to the object to be annotated, wherein the close-up annotation result includes a core area of ​​the object to be annotated; When the ratio is less than a preset value, a distant annotation result corresponding to the object to be annotated is determined, and the distant annotation result includes the entire area of ​​the object to be annotated.

7. The method according to claim 1, characterized in that The method further comprises: According to the obstacle recognition result corresponding to the target image, determining the avoidance strategy of the unmanned agricultural machinery from a preset strategy library; Based on the avoidance strategy, the path of the unmanned agricultural machine is replanned.

8. A device for detecting obstacles in unstructured farmland, characterized in that: include: An image acquisition module, used for acquiring a target image of an unstructured farmland where the unmanned agricultural machine is located based on at least one camera deployed on the unmanned agricultural machine; A feature extraction module, used to input the target image into an extraction network of an obstacle detection model, extract a multi-scale feature map of the target image, and respectively extract a channel attention map and a spatial attention map from the extracted feature map; A feature fusion module is used to use the fusion network of the obstacle detection model to perform a first fusion operation on the channel attention map and the spatial attention map corresponding to the same feature map to obtain a fused feature map, and to perform a second fusion operation on the fused feature maps or feature maps of different scales to obtain output features; The obstacle detection module is used to input the output features into the detection network of the obstacle detection model to obtain the obstacle recognition result corresponding to the target image.

9. An electronic device, characterized in that: include: processor; as well as A memory arranged to store computer executable instructions, the executable instructions being configured to be executed by the processor, the executable instructions comprising instructions for performing the steps in the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is used to store computer-executable instructions, and the executable instructions enable a computer to execute the method according to any one of claims 1 to 7.

Images