Agricultural robot control method, device and equipment and storage medium

Abstract

The invention discloses an agricultural robot control method, apparatus and device, and a storage medium. The method comprises the steps of obtaining a farmland environment image collected by an image collection device, and a basic attribute, a driving speed and a driving direction of an agricultural robot; inputting the farmland environment image into a pre-trained U-shaped semantic segmentation model based on a channel and a space attention mechanism, and outputting a feature image of a crop row in the farmland environment image; determining position information of the crop row closest to wheels of the agricultural robot according to the wheel track and the spatial position; converting the position information to a robot coordinate system according to the spatial position; the steering angular velocity of the agricultural robot is determined according to the position information converted to the robot coordinate system and the current running speed, and the agricultural robot is controlled to control the course angle of the robot according to the steering angular velocity. By means of the method, the operation efficiency of the agricultural robot is remarkably improved through the automatic crop row identification and control technology.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of automatic control technology, and in particular to an agricultural robot control method, device, equipment and storage medium. Background Art

[0002] At present, crop row identification and tracking control is a key technical link in agricultural automation. Crop rows are usually rows of plants planted in farmland. Maintaining a good regular arrangement of crop row clusters is the basis for efficient agricultural management. In order to improve production efficiency and reduce costs, modern agriculture has gradually introduced automated equipment such as unmanned agricultural machinery and field management robots. These devices need to rely on advanced computer vision and control technology to accurately identify and track crop row clusters, thereby realizing precision agricultural operations such as automatic weeding, spraying, fertilizing and harvesting.

[0003] Satellite navigation systems play a leading role in the navigation and positioning of autonomous agricultural robots, and can achieve decimeter-level or centimeter-level positioning accuracy. Especially for large-scale field planting patterns in large, centralized open farms, satellite signals are relatively stable and reliable, and satellite positioning technology is relatively mature. However, with the changes in the farmland environment, especially in the face of small-scale, decentralized, and regionalized farmland environments, satellite positioning navigation cannot automatically identify crop row clusters, and cannot plan paths, move, and perform corresponding field management operations based on the distance between the wheels and the crop rows, and cannot meet actual application needs. Summary of the invention

[0004] The embodiments of the present invention provide an agricultural robot control method, device, equipment and storage medium, which can reduce the dependence on manpower and significantly improve the operating efficiency of the agricultural robot through automatic identification of crop row clusters and tracking control technology.

[0005] In a first aspect, an embodiment of the present invention provides an agricultural robot control method, comprising:

[0006] Acquire farmland environment images acquired by an image acquisition device and basic attributes, driving speed and driving direction of the agricultural robot; the basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images according to a fixed acquisition angle;

[0007] The farmland environment image is input into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and a feature image of the crop rows in the farmland environment image is output; wherein the training data set of the U-shaped semantic segmentation model is the original farmland environment image collected under different environments; the crop rows and the positions of the crop rows are marked in the original farmland environment image;

[0008] Determine the position information of the crop row closest to the wheel of the agricultural robot according to the wheelbase and the spatial position;

[0009] Converting the position information into a robot coordinate system according to the spatial position;

[0010] The steering angular velocity of the agricultural robot is determined according to the position information converted into the robot coordinate system and the driving speed, and the agricultural robot is controlled to control the heading angle of the agricultural robot according to the steering angular velocity.

[0011] In a second aspect, an embodiment of the present invention further provides an agricultural robot control device, the device comprising:

[0012] An information acquisition module is used to acquire farmland environment images acquired by an image acquisition device and basic attributes, driving speed and driving direction of the agricultural robot; the basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images according to a fixed acquisition angle;

[0013] A feature image output module is used to input the farmland environment image into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and output a feature image of the crop rows in the farmland environment image; wherein the training data set of the U-shaped semantic segmentation model is the original farmland environment image collected under different environments; the crop rows and the positions of the crop rows are marked in the original farmland environment image;

[0014] A position information module, used to determine the position information of the crop row closest to the wheel of the agricultural robot according to the wheelbase and the spatial position;

[0015] A coordinate system conversion module, used for converting the position information into a robot coordinate system according to the spatial position;

[0016] A steering control module is used to determine the steering angular velocity of the agricultural robot according to the position information converted into the robot coordinate system and the driving speed, and control the agricultural robot to control the heading angle of the agricultural robot according to the steering angular velocity.

[0017] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0018] one or more processors;

[0019] a storage device for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the agricultural robot control method provided by the embodiment of the present disclosure.

[0021] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to implement the agricultural robot control method provided by the embodiments of the present disclosure.

[0022] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the agricultural robot control method provided in the embodiment of the first aspect above.

[0023] The present invention discloses an agricultural robot control method, device, equipment and storage medium, comprising: obtaining a farmland environment image acquired by an image acquisition device and basic attributes, driving speed and driving direction of the agricultural robot; the basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images according to a fixed acquisition angle; the farmland environment image is input into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and a feature image of a crop row in the farmland environment image is output; wherein the training data set of the U-shaped semantic segmentation model is an original farmland environment image acquired under different environments; the crop row and the position of the crop row are marked in the original farmland environment image; the position information of the crop row closest to the wheel of the agricultural robot is determined according to the wheelbase and the spatial position; the position information is converted into a robot coordinate system according to the spatial position; the steering angular velocity of the agricultural robot is determined according to the position information converted into the robot coordinate system and the driving speed, and the agricultural robot is controlled to control the heading angle of the agricultural robot according to the steering angular velocity. By using this method: through automatic identification of crop row clusters and tracking control technology, the dependence on manpower can be reduced and the operating efficiency of agricultural robots can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0025] Figure 1 A flowchart of an agricultural robot control method provided by an embodiment of the present disclosure;

[0026] Figure 2An example diagram of a characteristic image of an agricultural robot control method provided by an embodiment of the present disclosure;

[0027] Figure 3 A schematic diagram of the structure of an agricultural robot control device provided by an embodiment of the present disclosure;

[0028] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0036] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0037] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0039] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0040] Figure 1 A flowchart of an agricultural robot control is provided in an embodiment of the present disclosure. The embodiment of the present disclosure is suitable for providing a solution to the problem that satellite positioning navigation cannot automatically identify crop row clusters and cannot plan paths according to the distance between the wheels and the crop rows. The method can be executed by an agricultural robot control device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0041] like Figure 1 As shown, an agricultural robot control method provided by an embodiment of the present disclosure may specifically include the following steps:

[0042] S110, acquiring farmland environment images acquired by image acquisition equipment and basic attributes, driving speed, and driving direction of the agricultural robot.

[0043] Among them, the basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images according to a fixed acquisition angle.

[0044] In this embodiment, the image acquisition device may be a device for acquiring images, such as a camera installed on the agricultural robot. The basic attribute may be an attribute of the agricultural robot itself. The wheelbase may be the wheelbase of the vehicle or the distance between the left and right tracks of a tracked vehicle.

[0045] In this embodiment, the farmland environment image captured by the image acquisition device and the basic attributes, driving speed and driving direction of the agricultural robot are obtained. The basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot, and the image acquisition device captures images according to a fixed acquisition angle.

[0046] S120, inputting the farmland environment image into a pre-trained U-shaped semantic segmentation model based on channel and spatial attention mechanism, and outputting a feature image of the crop rows in the farmland environment image.

[0047] The training data set of the U-type semantic segmentation model is the original farmland environment images collected under different environments. The original farmland environment images are marked with crop rows and the locations of crop rows. The U-type semantic segmentation model includes: an encoder-decoder module and an output layer.

[0048] In this embodiment, the U-type semantic segmentation model of the channel and spatial attention mechanism is a VC-UNet semantic segmentation model improved based on the U-Net semantic segmentation model. The training data set is a robot image acquisition platform moving across crop rows and driving along crop rows, and the camera continuously collects crop images. For different weather light levels, shadow occlusion, and mixed crop row terrain environments, more than 3,500 data are collected, including six special cases such as strong light on sunny days, weak light on cloudy days, balanced light, horizontal crop ridges, vertical crop ridges, and shadow occlusion. By collecting the crop row data set and dividing it into a training set and a validation set at a ratio of 9:1, 3,150 training sets and 350 validation sets are selected. In order to ensure the consistency of the segmented data set with the original image, each crop row of the crop image is annotated. According to the segmentation target to be extracted, the annotation method uses line labels to annotate the starting and end points of the crop row and connect the points into lines.

[0049] Specifically,

[0050] Figure 2 This is an example diagram of a characteristic image of an agricultural robot control method provided in an embodiment of the present disclosure.

[0051] Based on the above embodiment, inputting the farmland environment image into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and outputting a feature image of a crop row in the farmland environment image may include the following steps:

[0052] a1) Inputting the farmland environment image into the encoder-decoder module, outputting the edge feature image of the crop rows in the farmland environment image and inputting the edge feature image into the output layer; the size of the edge feature image is the same as that of the farmland environment image.

[0053] b1) Compare the characteristic value in the edge feature image with the set threshold. When the characteristic value is greater than or equal to the set threshold, set the pixel point of the farmland environment image corresponding to the characteristic value to the first set value. When the characteristic value is less than the set threshold, set the pixel point of the farmland environment image corresponding to the characteristic value to the second set value, and output the feature image.

[0054] Specifically, the encoder and decoder in the encoder-decoder module. The main function of the encoder is to extract the features of the input image through the convolution layer and reduce the size of the feature map through the pooling layer, thereby reducing the amount of calculation and capturing higher-level semantic information. An encoder block usually contains two 3x3 convolution layers, a ReLU activation function, and a 2x2 maximum pooling layer to gradually reduce the spatial dimension of the feature map and increase the number of channels. Each layer of the encoder doubles the number of channels of the feature map and halves the resolution until the intermediate feature map is reached. The intermediate feature map is the last layer of the encoder part, which contains the highest level of feature representation extracted from the input image. These features will be used in the decoder part to reconstruct the structural information of the original image. Decoder: The function of the decoder is opposite to that of the encoder. It gradually restores the size of the feature map through upsampling operations and fuses multi-scale features from the encoder through jump connections to reconstruct the detailed information of the image. Each layer of the decoder halves the number of channels of the feature map and doubles the resolution until it reaches the same size as the input image. The skip connection mechanism allows the decoder to directly access the feature maps of the corresponding layer of the encoder during the upsampling process, which helps to preserve more detail information and improve the accuracy of segmentation.

[0055] In this embodiment, the first set value and the second set value are preset values, which are set according to actual conditions and are not specifically limited in this embodiment. For example, the first set value may be 0 and the second set value may be 255.

[0056] Specifically, the output layer is usually a 1x1 convolution layer, which is used to map the feature map generated by the decoder to the final segmentation result. The feature value in the edge feature image is compared with the set threshold. When the feature value is greater than or equal to the set threshold, the pixel point of the farmland environment image corresponding to the feature value is set to the first set value. When the feature value is less than the set threshold, the pixel point of the farmland environment image corresponding to the feature value is set to the second set value, and the feature image is output. The number of output channels of this convolution layer is usually equal to the number of categories in the image, for example, 2 in a binary classification problem.

[0057] S130. Determine the position information of the crop row closest to the wheel of the agricultural robot according to the wheelbase and the spatial position.

[0058] In an embodiment, the location information may be a function mapping relationship corresponding to the crop rows meeting the set conditions.

[0059] Specifically, the wheel position information of the wheel in the feature image is determined according to the wheelbase and the spatial position; a corresponding rectangular area is selected from the feature image as a region of interest according to preset coordinates; the preset coordinates are the coordinates of the four corners of the rectangular area; and a function mapping relationship corresponding to the crop row in the region of interest is extracted;

[0060] Determine the distance between each function mapping relationship and the wheel position information; and use a set number of function mapping relationships with the smallest distance as the position information of the corresponding crop row.

[0061] Based on the above embodiment, determining the position information of two crop rows whose sum of distances from the center point of the characteristic image to the two crop rows is greater than the wheelbase and whose sum of distances is the smallest may include the following steps:

[0062] a2) selecting a corresponding rectangular area from the feature image as the region of interest according to preset coordinates; the preset coordinates are the coordinates of the four corners of the rectangular area.

[0063] b2) Extract the function mapping relationship corresponding to the crop rows in the area of ​​interest.

[0064] c2) determining the distance between each function mapping relationship and the wheel position information.

[0065] d2) using the function mapping relationship of the set number with the smallest distance as the position information of the corresponding crop row.

[0066] In this embodiment, the function mapping relationship may be a straight line. The set number may be 1 or 2 in this embodiment. It is determined according to the ratio between the distance between the crop rows and the wheel spacing. When the ratio is 1, the set number is 1; when the ratio is greater than 1, the set number is 2.

[0067] Specifically, the function mapping relationship corresponding to the crop row in the region of interest is extracted by using Hough linear transformation, and the distance between each function mapping relationship and the wheel position information is determined. The function mapping relationships with the smallest set number of distances are used as the corresponding position information of the crop row. Among them, the distance is the smaller distance between each function mapping relationship and the wheel position information, and the threshold for setting the distance determination is combined to merge the straight lines with similar distances.

[0068] Based on the above embodiment, the region of interest is automatically adjusted. The driving speed is obtained in real time by the satellite positioning system to obtain the chassis forward speed in the vehicle body coordinate system, and the boundary box of the region of interest is updated by calculating the boundary value of the mask image.

[0069] On the basis of this embodiment, it also includes: sorting the merged i straight lines according to the absolute value distance to the center coordinate of the ROI cropping image, with serial numbers 1, 2, ..., i. Determine to extract one or two straight line equations based on the distance between the crop rows and the ratio q between the wheel spacing. If q is not an integer, it is rounded to the nearest integer. When q is 1, extract the straight line closest to the center coordinate, that is, the straight line equation with a sorting number of 1; when q is greater than 1, read the straight line equation with a sorting number of q and the straight line equation with a sorting number of q-1, and calculate the median of the two straight lines, and the median represents the center line of the crop strip. When the wheel spacing is equal to the strip spacing, q is 1.

[0070] S140, converting the position information into the robot coordinate system according to the spatial position.

[0071] Based on the above embodiment, converting the position information into the robot coordinate system according to the spatial position may include the following steps:

[0072] a3) determining a coordinate transformation matrix according to the spatial position and a known transformation formula from the robot coordinate system to the camera coordinate system.

[0073] b3) The multiplication result of the position information and the coordinate transformation matrix is ​​used as the position information in the robot coordinate system.

[0074] Specifically, the conversion matrix from pixels to physical units is determined according to the acquisition angle, installation height and installation position of the acquisition device on the robot, and then converted. It can also be calculated based on the mapping relationship between the two-dimensional image and the three-dimensional strip model.

[0075] S150. Determine a steering angular velocity of the agricultural robot according to the position information converted into the robot coordinate system and the driving speed, and control the agricultural robot to control the heading angle of the agricultural robot according to the steering angular velocity.

[0076] Specifically, the center line of the position information is used as the navigation center line. The current motion state information of the agricultural robot positioned by the positioning device is obtained. The angle deviation between the navigation center line and the driving direction and the lateral deviation between the navigation center line and the current motion state information are determined. The steering angular velocity is determined according to the angle deviation, the lateral deviation and the driving speed.

[0077] Based on the above embodiment, determining the steering angular velocity of the agricultural robot according to the position information, the driving speed and the driving direction converted into the robot coordinate system may include the following steps:

[0078] Agricultural robots also include positioning devices

[0079] a4) The center line of the two position information is used as the navigation center line.

[0080] b4) Acquire the current motion state information of the agricultural robot positioned by the positioning device.

[0081] In this embodiment, the current motion state information includes the current real-time position, measured vehicle speed and heading angle. If the positioning device has no signal, the position, speed, heading and other motion state information published by the chassis odometer is used.

[0082] c4) determining an angular deviation between the navigation centerline and the current motion state information and a lateral deviation between the navigation centerline and the current motion state information.

[0083] d4) Determine the steering angle velocity based on the angular deviation, lateral deviation and driving speed.

[0084] Specifically, calculate the angle deviation ang in the image coordinate system t

[0085] ang t =arctan(k L )

[0086] Calculate the lateral deviation dis in the image coordinate system t

[0087]

[0088] Among them, k L represents the slope of the extracted navigation centerline; b L Represents the intercept of the extracted navigation centerline, img h represents the pixel width of the image; ω represents the set speed adjustment coefficient; v represents the driving speed of the agricultural robot. Convert the angular deviation and lateral deviation in the image coordinate system to the robot coordinate system

[0089] The calculation formula for determining the steering angular velocity based on the angle deviation, lateral deviation and driving speed is as follows:

[0090]

[0091] Where, δ represents the steering angular velocity; L represents the wheelbase; ang e Indicates the angle deviation in the robot coordinate system; eRepresents the lateral deviation in the robot coordinate system; η represents the first gain coefficient of the controller; γ represents the second gain coefficient of the controller: v represents the driving speed.

[0092] Based on the above embodiment, it also includes:

[0093] a5) dividing the feature image into a plurality of sub-block areas according to a sliding window of a set size; the sliding window processes the feature images in sequence from top to bottom;

[0094] b5) determining the pixel sum value of each row in the sub-block area;

[0095] c5) when there is a pixel sum value greater than a set threshold, the position of the pixel sum value is the position of the end of the crop row;

[0096] d5) When the position of the end of the crop row is within the set range, the agricultural robot will perform an exit action.

[0097] Specifically, the feature image is cut according to a sliding window of a set size to obtain multiple sub-block areas; the sliding window processes each sub-block area from top to bottom; the pixel sum value of each row in the sub-block area where the sliding window is located is determined; when there is a pixel sum value greater than the set threshold, it is determined that the agricultural robot has reached the end of the crop row. When the position of the end of the crop row is within the set range, the robot slows down and performs an exit action. Usually, when the crop row end position value returned by the EDR function is detected in a sub-area within the range of [2h, 3h], the robot will perform an exit action, where h is the height of the region of interest (ROI).

[0098] Based on the above embodiment, the present invention further includes: determining the current position EDR of the agricultural robot from the end of the crop row:

[0099]

[0100] I(x,Y) is the predicted value of the semantic segmentation model, Indicates the calculation of the pixel cumulative value of the sub-block area; Y represents the vertical coordinate of the sub-block area; H is the width of the sub-block area. After the valid EDR value is detected for the first time, a mean filter is used for filtering to reduce the noise of the row end position value EDR.

[0101] Speed ​​adjustment formula for agricultural robots:

[0102]

[0103] v cmd : The target speed of the robot, i.e. the control instruction speed, v max: the maximum permissible speed of the robot; EDR: the position of the agricultural robot from the end of the crop row; h: the maximum height of the sub-block area; δ: the current turning angular velocity of the robot, δ max : Maximum angular velocity allowed when turning; α: Turning deceleration coefficient, which determines the speed drop when turning; β: Controls the speed of distance attenuation, which determines the speed drop to the end of the crop row.

[0104] The invention discloses an agricultural robot control method, comprising: acquiring a farmland environment image acquired by an image acquisition device and basic attributes, a driving speed and a driving direction of the agricultural robot; the basic attributes include: a wheelbase and a spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images at a fixed acquisition angle; the farmland environment image is input into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and a feature image of a crop row in the farmland environment image is output; wherein the training data set of the U-shaped semantic segmentation model is an original farmland environment image acquired under different environments; the crop rows and the positions of the crop rows are marked in the original farmland environment image; the position information of the crop row closest to the wheel of the agricultural robot is determined according to the wheelbase and the spatial position; the position information is converted into a robot coordinate system according to the spatial position; the steering angular velocity of the agricultural robot is determined according to the position information converted into the robot coordinate system and the driving speed, and the agricultural robot is controlled to control the heading angle of the agricultural robot according to the steering angular velocity. By using this method: through automatic identification of crop row clusters and tracking control technology, the dependence on manpower can be reduced and the operating efficiency of agricultural robots can be significantly improved.

[0105] Figure 3 The present invention also provides a schematic diagram of the structure of an agricultural robot control device, as shown in FIG. Figure 3 As shown, the device includes: an information acquisition module 210, a feature image output module 220, a position information module 230, a coordinate system conversion module 240 and a control steering module 250.

[0106] The information acquisition module 210 is used to acquire the farmland environment image acquired by the image acquisition device and the basic attributes, driving speed and driving direction of the agricultural robot; the basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images according to a fixed acquisition angle;

[0107] The feature image output module 220 is used to input the farmland environment image into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and output a feature image of the crop rows in the farmland environment image; wherein the training data set of the U-shaped semantic segmentation model is the original farmland environment image collected under different environments; the crop rows and the positions of the crop rows are marked in the original farmland environment image;

[0108] A position information module 230, configured to determine position information of the crop row closest to the wheel of the agricultural robot according to the wheelbase and the spatial position;

[0109] A coordinate system conversion module 240, used to convert the position information into a robot coordinate system according to the spatial position;

[0110] The steering control module 250 is used to determine the steering angular velocity of the agricultural robot according to the position information converted to the robot coordinate system and the driving speed, and control the heading angle of the agricultural robot according to the steering angular velocity.

[0111] The technical solution provided by the embodiments of the present disclosure can reduce dependence on manpower and significantly improve the operating efficiency of agricultural robots through automatic identification of crop row clusters and tracking control technology.

[0112] Furthermore, the feature image output module 220 may be used to:

[0113] The U-shaped semantic segmentation model includes: an encoder-decoder module and an output layer;

[0114] Input the farmland environment image into the encoder-decoder module, output the edge feature image of the crop row in the farmland environment image and input the edge feature image into the output layer; the size of the edge feature image is the same as the size of the farmland environment image;

[0115] The characteristic value in the edge feature image is compared with a set threshold. When the characteristic value is greater than or equal to the set threshold, the pixel point of the farmland environment image corresponding to the characteristic value is set to a first set value. When the characteristic value is less than the set threshold, the pixel point of the farmland environment image corresponding to the characteristic value is set to a second set value, and the feature image is output.

[0116] Furthermore, the location information module 230 may be used to:

[0117] Determining wheel position information of the wheel in the feature image according to the wheel width and the spatial position;

[0118] Selecting a corresponding rectangular area from the feature image as a region of interest according to preset coordinates; the preset coordinates are the coordinates of the four corners of the rectangular area;

[0119] Extracting the function mapping relationship corresponding to the crop row in the region of interest;

[0120] Determine the distance between each function mapping relationship and the wheel position information;

[0121] The function mapping relationship of the set number with the smallest distance is used as the position information of the corresponding crop row.

[0122] Furthermore, the location determination module 240 may be used to:

[0123] Determine a coordinate transformation matrix according to the spatial position and a known transformation formula from the robot coordinate system to the camera coordinate system;

[0124] The multiplication result of the position information and the coordinate transformation matrix is ​​used as the position information in the robot coordinate system.

[0125] Furthermore, the control steering module 250 may be used to:

[0126] The agricultural robot also includes a positioning device; the center line of the two position information is used as a navigation center line;

[0127] Acquiring current motion state information of the agricultural robot positioned by the positioning device;

[0128] Determining an angular deviation between the navigation centerline and the current motion state information and a lateral deviation between the navigation centerline and the current motion state information;

[0129] The steering angle speed is determined as a function of the angular deviation, the lateral deviation, and the driving speed.

[0130] Furthermore, the location determination module 240 may be used to:

[0131] The calculation formula for determining the steering angular velocity according to the angle deviation, the lateral deviation and the driving speed is as follows:

[0132]

[0133] Wherein, δ represents the steering angular velocity; L represents the wheelbase; ang e Indicates the angle deviation; e represents the lateral deviation; η represents the first gain coefficient of the controller; γ represents the second gain coefficient of the controller: v represents the driving speed.

[0134] Furthermore, the device also includes:

[0135] The characteristic image is divided into a plurality of sub-block areas according to a sliding window of a set size; the sliding window processes the characteristic image in sequence from top to bottom;

[0136] Determine the pixel sum value of each row in the sub-block area;

[0137] When there is a pixel sum value greater than a set threshold, the position of the pixel sum value is the position of the end of the crop row;

[0138] When the position of the end of the crop row is within the set range, the agricultural robot will perform an exit action.

[0139] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0140] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is given. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0141] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0142] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the agricultural robot control method.

[0144] In some embodiments, the agricultural robot control method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the agricultural robot control method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the agricultural robot control method in any other appropriate manner (e.g., by means of firmware).

[0145] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0147] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for controlling an agricultural robot, characterized in that: include: Obtain farmland environment images collected by image acquisition equipment and basic attributes and driving speed of the agricultural robot; The basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images according to a fixed acquisition angle; The farmland environment image is input into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and a feature image of the crop rows in the farmland environment image is output; wherein the training data set of the U-shaped semantic segmentation model is the original farmland environment image collected under different environments; the crop rows and the positions of the crop rows are marked in the original farmland environment image; Determine the position information of the crop row closest to the wheel of the agricultural robot according to the wheelbase and the spatial position; Converting the position information into a robot coordinate system according to the spatial position; The steering angular velocity of the agricultural robot is determined according to the position information converted into the robot coordinate system and the driving speed, and the agricultural robot is controlled to control the heading angle of the agricultural robot according to the steering angular velocity.

2. The method according to claim 1, characterized in that The U-shaped semantic segmentation model includes: an encoder-decoder module and an output layer; accordingly, the farmland environment image is input into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and a feature image of a crop row in the farmland environment image is output, including: Input the farmland environment image into the encoder-decoder module, output the edge feature image of the crop row in the farmland environment image and input the edge feature image into the output layer; the size of the edge feature image is the same as the size of the farmland environment image; The characteristic value in the edge feature image is compared with a set threshold. When the characteristic value is greater than or equal to the set threshold, the pixel point of the farmland environment image corresponding to the characteristic value is set to a first set value. When the characteristic value is less than the set threshold, the pixel point of the farmland environment image corresponding to the characteristic value is set to a second set value, and the feature image is output.

3. The method according to claim 1, characterized in that The step of determining the position information of the crop row closest to the wheel of the agricultural robot according to the wheelbase and the spatial position includes: Determining wheel position information of the wheel in the feature image according to the wheel width and the spatial position; Selecting a corresponding rectangular area from the feature image as a region of interest according to preset coordinates; the preset coordinates are the coordinates of the four corners of the rectangular area; Extracting the function mapping relationship corresponding to the crop row in the region of interest; Determine the distance between each function mapping relationship and the wheel position information; The function mapping relationship of the set number with the smallest distance is used as the position information of the corresponding crop row.

4. The method according to claim 1, characterized in that: The converting the position information into a robot coordinate system according to the spatial position includes: Determine a coordinate transformation matrix according to the spatial position and a known transformation formula from the robot coordinate system to the camera coordinate system; The multiplication result of the position information and the coordinate transformation matrix is ​​used as the position information in the robot coordinate system.

5. The method according to claim 1, characterized in that The agricultural robot further includes a positioning device, and correspondingly, the steering angular velocity of the agricultural robot is determined according to the position information and the driving speed converted into the robot coordinate system, including: Using the center line of the position information as the navigation center line; Acquiring current motion state information of the agricultural robot positioned by the positioning device; Determining an angular deviation between the navigation centerline and the current motion state information and a lateral deviation between the navigation centerline and the current motion state information; The steering angle speed is determined as a function of the angular deviation, the lateral deviation, and the driving speed.

6. The method according to claim 5, characterized in that The calculation formula for determining the steering angular velocity according to the angle deviation, the lateral deviation and the driving speed is as follows: Wherein, δ represents the steering angular velocity; L represents the wheelbase; ang e Indicates the angle deviation; e represents the lateral deviation; η represents the first gain coefficient of the controller; γ represents the second gain coefficient of the controller: v represents the driving speed.

7. The method according to claim 1, characterized in that Also includes: Dividing the feature image into a plurality of sub-block areas according to a sliding window of a set size; The sliding window processes the feature images sequentially from top to bottom; Determine the pixel sum value of each row in the sub-block area; When there is a pixel sum value greater than a set threshold, the position of the pixel sum value is the position of the end of the crop row; When the position of the end of the crop row is within the set range, the agricultural robot will perform an exit action.

8. An agricultural robot control device, characterized in that: include: An information acquisition module is used to obtain farmland environment images collected by image acquisition equipment and basic attributes, driving speed and driving direction of the agricultural robot; The basic attributes include: wheelbase and the spatial position of the image acquisition device on the agricultural robot; the image acquisition device acquires images according to a fixed acquisition angle; A feature image output module is used to input the farmland environment image into a pre-trained U-shaped semantic segmentation model based on a channel and spatial attention mechanism, and output a feature image of the crop rows in the farmland environment image; wherein the training data set of the U-shaped semantic segmentation model is the original farmland environment image collected under different environments; the crop rows and the positions of the crop rows are marked in the original farmland environment image; A position information module, used to determine the position information of the crop row closest to the wheel of the agricultural robot according to the wheelbase and the spatial position; A coordinate system conversion module, used for converting the position information into a robot coordinate system according to the spatial position; A steering control module is used to determine the steering angular velocity of the agricultural robot according to the position information converted into the robot coordinate system and the driving speed, and control the agricultural robot to control the heading angle of the agricultural robot according to the steering angular velocity.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the agricultural robot control method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the agricultural robot control method described in any one of claims 1-7 when executed.

Images