Visual navigation method, system, device and medium for servo robot

Abstract

This application relates to a visual navigation method, system, device, and medium for a servo robot, applied to the field of robot navigation technology. The method includes obtaining panoramic image data and image scale data of a target farmland; constructing a three-dimensional farmland model of the target farmland based on the panoramic image data and image scale data; obtaining crop identification data and weed identification data of the target farmland, a servo robot's peripheral visual image, and robot position information; generating a three-dimensional plant model based on the crop identification data and weed identification data; constructing a working farmland environment model based on the three-dimensional farmland model and the three-dimensional plant model; and constructing a visual navigation route for the servo robot based on the peripheral visual image, robot position information, and the working farmland environment model. This application has the effect of enabling precise navigation and control of the robot based on actual environmental characteristics.

Description

Technical Field

[0001] The present application relates to the technical field of robot navigation, and in particular to a visual navigation method, system, device and medium for a servo robot. Background Art

[0002] With the development of modern agriculture, robots are increasingly used in farmland management, especially playing an important role in operations such as weeding, sowing, and fertilizing.

[0003] However, precise robot navigation in complex farmland environments still faces numerous challenges. Traditional navigation methods rely primarily on GPS or inertial navigation systems, but these technologies often suffer from insufficient accuracy and poor adaptability in densely planted farmland environments. For example, GPS signals are susceptible to interference when obscured by vegetation, while inertial navigation can lead to positioning errors due to accumulated errors, affecting the accuracy and efficiency of robot operations. When using existing navigation methods for weeding, inaccurate navigation can easily lead to missed weeds or damage to plants.

[0004] Therefore, there is an urgent need for a technology that can accurately navigate and control robots based on the actual environment characteristics. Summary of the Invention

[0005] In order to enable precise navigation and control of a robot according to actual environmental characteristics, the present application provides a visual navigation method, system, device and medium for a servo robot.

[0006] In a first aspect, the present application provides a visual navigation method for a servo robot, which adopts the following technical solution:

[0007] A visual navigation method for a servo robot, comprising:

[0008] Obtain panoramic image data and image scale data of the target farmland;

[0009] constructing a three-dimensional farmland model of the target farmland based on the panoramic picture data and the image scale data;

[0010] Acquiring crop identification data, weed identification data, a peripheral visual image of the servo robot, and robot position information of the target farmland;

[0011] generating a three-dimensional plant model based on the crop identification data and the weed identification data;

[0012] Constructing a working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model;

[0013] A visual navigation route of the servo robot is constructed based on the peripheral visual image, the robot position information and the working farmland environment model.

[0014] By adopting the above technical solution, a three-dimensional model of the target farmland is constructed using the captured panoramic image data and image scale data, so that the constructed three-dimensional model is consistent with the actual target farmland, and a plant three-dimensional model is constructed based on the identified crop identification data and weed identification data. The two three-dimensional models are then combined to ensure that the plant model and farmland model in the final working farmland environment model are more accurate. The visual navigation route is formulated by combining the surrounding visual images and the robot's position information, so that the formulated route is not only more in line with the actual farmland environment, but also more accurate, so that the robot can be precisely navigated and controlled according to the actual environmental characteristics.

[0015] Optionally, constructing the three-dimensional farmland model of the target farmland based on the panoramic picture data and the image scale data includes:

[0016] Determining the regional image and planting position of the target farmland based on the panoramic image data;

[0017] Matching the regional image with a farmland model in a preset model database to determine the regional similarity between the regional image and the farmland model;

[0018] Determining whether the farmland model needs to be morphologically adjusted based on the regional similarity;

[0019] If the farmland model needs to be morphologically adjusted, determining the regional image to adjust the farmland model to generate an adjusted farmland model;

[0020] constructing a three-dimensional farmland model based on the image scale data, the adjusted farmland model, and the planting positions;

[0021] If the farmland model does not need to be morphologically adjusted, a three-dimensional farmland model is constructed based on the image scale data, the farmland model and the planting positions.

[0022] Optionally, generating a three-dimensional plant model based on the crop identification data and the weed identification data includes:

[0023] determining species morphological difference information based on the crop identification data and the weed identification data;

[0024] determining crop locations based on the crop identification data, and determining weed locations based on the weed identification data;

[0025] adding the crop identification data and the weed identification data to the planting location according to the crop location and the weed location to generate a plant model;

[0026] Based on the species morphological difference information and preset identification rules, feature identification is added to the plant model to generate a three-dimensional plant model.

[0027] Optionally, constructing a working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model includes:

[0028] Acquiring plant size data of the plant three-dimensional model;

[0029] Calculating farmland size data of the three-dimensional farmland model based on the panoramic picture data and the image scale data;

[0030] Calculating the location size of the planting location based on the farmland size data;

[0031] Calculating a size ratio of the three-dimensional plant model based on the position size and the plant size data;

[0032] Adjusting the plant three-dimensional model based on the size ratio to generate an adjusted plant model;

[0033] The adjusted plant model is fused with the farmland three-dimensional model according to the planting position to construct a working farmland environment model.

[0034] Optionally, constructing a visual navigation route for the servo robot based on the peripheral visual image, the robot position information, and the working farmland environment model includes:

[0035] Obtaining the endurance information of the servo robot;

[0036] generating a first navigation route based on the endurance information and the working farmland environment model;

[0037] Dividing the working farmland environment model into key weeding areas based on feature identifiers in the three-dimensional plant model, and generating a division result;

[0038] adjusting the first navigation route based on the division result to generate an adjusted route;

[0039] Adjusting the position of the servo robot based on the peripheral visual image, the robot position information, and the adjustment route to determine an initial position and an end position;

[0040] A visual navigation route is generated based on the initial position, the end position, and the adjusted route.

[0041] Optionally, adjusting the first navigation route based on the division result to generate the adjusted route includes:

[0042] Determine the regional proportion and regional location of the key weed control areas based on the division results;

[0043] Determining whether the servo robot is capable of single serial processing based on the area proportion and the area position;

[0044] If the servo robot is capable of single serial processing, a focused weeding route is generated based on the location of the area;

[0045] Adjusting the first navigation route based on the key weeding route, the regional location, and the endurance information to generate an adjusted route;

[0046] If the servo robot cannot process the process in a single series, the first navigation route is adjusted based on the area proportion, the area position and the endurance information to generate an adjusted route.

[0047] Optionally, after constructing the visual navigation route of the servo robot based on the peripheral visual image, the robot position information and the working farmland environment model, the method further includes:

[0048] determining a predicted position of a first weed based on the initial position, the working farmland environment model, and the visual navigation route;

[0049] acquiring visual image data and plant position data when the servo robot reaches the initial position;

[0050] determining whether the predicted position is accurate based on the initial position, the visual image data, and the plant position data;

[0051] If the predicted position is accurate, weeding navigation is performed based on the visual navigation route and the working farmland environment model;

[0052] If the predicted position is inaccurate, the predicted positions of all plants are corrected based on the image data and the plant position data to generate a corrected position;

[0053] Weeding navigation is performed based on the corrected position, the visual navigation route, and the working farmland environment model.

[0054] In a second aspect, the present application provides a visual navigation system for a servo robot, which adopts the following technical solution:

[0055] A visual navigation system for a servo robot, comprising:

[0056] The farmland data acquisition module is used to obtain panoramic image data and image scale data of the target farmland;

[0057] A farmland model construction module, configured to construct a three-dimensional farmland model of the target farmland based on the panoramic image data and the image scale data;

[0058] a crop information acquisition module, configured to acquire crop identification data, weed identification data, a peripheral visual image of the servo robot, and robot position information of the target farmland;

[0059] a plant model construction module, configured to generate a three-dimensional plant model based on the crop identification data and the weed identification data;

[0060] An environment model construction module, configured to construct a working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model;

[0061] A navigation route construction module is used to construct a visual navigation route for the servo robot based on the peripheral visual image, the robot position information and the working farmland environment model.

[0062] By adopting the above technical solution, a three-dimensional model of the target farmland is constructed using the captured panoramic image data and image scale data, so that the constructed three-dimensional model is consistent with the actual target farmland, and a plant three-dimensional model is constructed based on the identified crop identification data and weed identification data. The two three-dimensional models are then combined to ensure that the plant model and farmland model in the final working farmland environment model are more accurate. The visual navigation route is formulated by combining the surrounding visual images and the robot's position information, so that the formulated route is not only more in line with the actual farmland environment, but also more accurate, so that the robot can be precisely navigated and controlled according to the actual environmental characteristics.

[0063] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0064] An electronic device comprising a processor coupled to a memory;

[0065] The processor is used to execute the computer program stored in the memory, so that the electronic device executes the computer program of the visual navigation method for a servo robot as described in any one of the first aspects.

[0066] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0067] A computer-readable storage medium stores a computer program that can be loaded by a processor and executes the visual navigation method for a servo robot according to any one of the first aspects.

[0068] In summary, this application includes at least one of the following beneficial technical effects:

[0069] A three-dimensional model of the target farmland is constructed using the captured panoramic image data and image scale data, so that the constructed three-dimensional model is consistent with the actual target farmland. A three-dimensional plant model is constructed based on the identified crop identification data and weed identification data. The two three-dimensional models are then combined to ensure that the plant model and farmland model in the final working farmland environment model are more accurate. The visual navigation route is formulated by combining the surrounding visual images and the robot's position information, so that the formulated route is not only more in line with the actual farmland environment, but also more accurate, so that the robot can be precisely navigated and controlled according to the actual environmental characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flow chart of a visual navigation method for a servo robot provided in an embodiment of the present application.

[0071] Figure 2 This is a schematic diagram showing plant information provided in an embodiment of the present application.

[0072] Figure 3 Schematic diagram of a three-dimensional plant model provided in an embodiment of the present application.

[0073] Figure 4 This is a structural block diagram of a visual navigation system for a servo robot provided in an embodiment of the present application.

[0074] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The present application is further described in detail below with reference to the accompanying drawings.

[0076] The present invention provides a visual navigation method for a servo robot. The visual navigation method for a servo robot can be executed by an electronic device, which can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, a tablet computer, a desktop computer, etc., but is not limited thereto.

[0077] Figure 1 A flowchart of a visual navigation method for a servo robot provided in an embodiment of the present application.

[0078] like Figure 1As shown, the main process of the method is described as follows (steps S101 to S106):

[0079] Step S101: Obtain panoramic image data and image scale data of a target farmland.

[0080] In this embodiment, the target farmland is an entire piece of farmland that needs to be weeded, such as a rice field, a corn field, and a valley. The panoramic image data can be image data including the entire farmland recorded after the planting is completed, or it can be image data collected by panoramic acquisition equipment such as drones before or after planting. It is necessary to ensure that the panoramic image data is consistent with the actual farmland allocation status when weeding is required, that is, it is necessary to ensure that the position of the furrows and ridges in the target farmland is consistent with the current state when shooting. The image ratio data is the size ratio of the image to the actual target farmland. It should be noted that this application will use corn fields as an example for full explanation.

[0081] Step S102 : constructing a three-dimensional farmland model of the target farmland based on the panoramic image data and the image scale data.

[0082] For step S102, the regional image and planting position of the target farmland are determined based on the panoramic image data; the regional image is matched with the farmland model in the preset model database to determine the regional similarity between the regional image and the farmland model; based on the regional similarity, it is judged whether the farmland model needs to be morphologically adjusted; if the farmland model needs to be morphologically adjusted, it is determined that the regional image is adjusted to the farmland model to generate an adjusted farmland model; a three-dimensional farmland model is constructed based on the image scale data, the adjusted farmland model and the planting position; if the farmland model does not need to be morphologically adjusted, a three-dimensional farmland model is constructed based on the image scale data, the farmland model and the planting position.

[0083] In this embodiment, after collecting the panoramic picture data and image ratio data, image analysis is performed on the panoramic picture data to determine the regional image and planting position of the target farmland, wherein the regional image of the target farmland is the overall shape of the target farmland, such as a rectangle, square or other regular or irregular shapes, and the planting position is the position for planting corn, that is, the position of the ridge in the target farmland. Both the regional image and the planting position can be directly obtained by performing graphic analysis on the panoramic picture data.

[0084] A three-dimensional model of the farmland is constructed based on the obtained regional image and the preset model database. The regional image is compared with the model shapes of all the farmland models recorded in the preset model database to find out whether there is a model shape consistent with the regional image. If there is a model shape consistent with the regional image, it will be determined that the farmland model corresponding to the model shape in the region is 100% similar to the regional image, and the farmland model does not need to be morphologically adjusted. The planting position can be directly added to the farmland model. After that, the three-dimensional model of the farmland is directly created according to the image scale data and the farmland model with the planting position added. If there is no model shape consistent with the regional image, a farmland model of the same type will be selected from the model shape according to the regional image, and the selected farmland model will be used as the standby farmland model. The differences between the regional image and the model shape will be compared, and the similarity between the regional image and the model shape will be calculated based on the differences and the preset similarity rules. The farmland model corresponding to the model shape with the highest similarity will be selected for model adjustment, that is, the model shape will be adjusted to a state consistent with the regional graphics, thereby obtaining an adjusted farmland model. The adjusted farmland model is a model consistent with the target farmland. The planting position will then be added to the adjusted farmland model, and then a three-dimensional farmland model will be created according to the image scale data and the adjusted farmland model with the added planting position.

[0085] It should be noted that when classifying the types of regional images and model shapes, including but not limited to circular, rectangular, polygonal and special-shaped types, the types are classified according to the characteristics of the regional images, and the model shapes are compared according to the division results to obtain the comparison results. Then, the regional image and the model shape are compared to outline the difference points. Then, the similarity between the regional image and the model shape is determined using the preset similarity calculation rules and the difference points. The preset similarity calculation rules include similarity deduction values ​​corresponding to different difference data such as the number of difference points and area. The similarity value can be calculated based on the sum of the similarity deduction values. The specific type and similarity deduction value need to be set according to the actual situation and are not specifically limited here.

[0086] Step S103 , obtaining crop identification data of the target farmland, weed identification data, the peripheral vision image of the servo robot, and the robot position information.

[0087] In this embodiment, the crop identification data includes the planting location and growth form of each corn seedling, and the weed identification data includes the growth location and growth form of each weed. The servo robot collects information about its surrounding environment to obtain a surrounding visual image, and calculates its own position through the various sensors installed on it and the manually set current position coordinates, and uses the obtained position as the robot's position information.

[0088] Step S104: generating a three-dimensional plant model based on the crop identification data and the weed identification data.

[0089] With respect to step S104, species morphological difference information is determined based on the crop identification data and the weed identification data; crop positions are determined based on the crop identification data, and weed positions are determined based on the weed identification data; the crop identification data and the weed identification data are added to the planting positions according to the crop positions and the weed positions to generate a plant model; and feature identifiers are added to the plant model based on the species morphological difference information and preset identification rules to generate a three-dimensional plant model.

[0090] In this embodiment, in order to make the final farmland environment model more accurate, the farmland three-dimensional model and the plant three-dimensional model are constructed separately. After the farmland three-dimensional model is created, the crop identification data and weed identification data collected in advance are processed to construct a plant three-dimensional model. The plant three-dimensional model includes the position, shape and arrangement of each plant, so that weeds in crops can be removed more accurately during weeding, and the possibility of accidental damage to crops can be reduced.

[0091] First, morphological difference information between crops and weeds, namely, height difference, color difference, and growth morphology difference, is determined based on morphological data in the crop identification data and the weed identification data. The obtained morphological difference information is used to perform feature identification addition processing together with preset representation rules. For example, weeds are marked with a special color, crops are marked with a special color, or both weeds and crops are marked with a special color. Since general crops and weeds are green plants, confusion is more likely to occur when the height of crops and weeds is similar, resulting in omissions in weeding or accidental damage to crops. Therefore, special color marking of one or both of them can effectively distinguish them, making it more accurate when setting the navigation route.

[0092] Before or after adding feature identification, the crop position and weed position can be determined based on the crop identification data and weed identification data. Each plant is added to the planting position according to the crop position and weed position to generate a plant model, and the feature identification is added to obtain the final plant three-dimensional model. It should be noted that the crop identification data and weed identification data both record the position information and the serial number of the field ridge where the plant is located, so that the plant can be accurately added to the planting position.

[0093] Step S105 : constructing a working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model.

[0094] For step S105, the plant size data of the plant three-dimensional model is obtained; the farmland size data of the farmland three-dimensional model is calculated based on the panoramic picture data and the image ratio data; the position size of the planting position is calculated based on the farmland size data; the size ratio of the plant three-dimensional model is calculated based on the position size and the plant size data; the plant three-dimensional model is adjusted based on the size ratio to generate an adjusted plant model; the adjusted plant model is merged with the farmland three-dimensional model according to the planting position to construct a working farmland environment model.

[0095] In this embodiment, after the farmland 3D model and the plant 3D model are constructed separately, the farmland 3D model and the plant 3D model are merged to obtain the final working farmland environment model. Since the sizes of the models were not unified when the models were created, there may be size inconsistencies. Before fusion, the sizes of the two models need to be adjusted to be consistent. During the adjustment, the farmland size data of the farmland 3D model is calculated using the panoramic image data and the image ratio data, that is, the overall size of the farmland 3D model is calculated, so that the spacing size of the planting position and the length, width, and thickness of the planting position itself can be obtained. The spacing size, length, width, and thickness are used as the position size. The size ratio of the plant 3D model can be obtained from the position size and plant size data. That is, the difference ratio of the position size in the farmland 3D model is calculated based on the plant size and position size in the plant 3D model. The difference ratio is used as the size ratio. The size ratio can be used to adjust the size of the plant 3D model to obtain an adjusted plant model. The adjusted plant model is merged into the farmland 3D model according to the planting position, thereby obtaining the final working farmland environment model. It should be noted that when the scale ratio is zero, the plant model will be directly integrated into the three-dimensional farmland model according to the planting position.

[0096] Step S106: constructing a visual navigation route for the servo robot based on the surrounding visual image, the robot position information, and the working farmland environment model.

[0097] For step S106, the endurance information of the servo robot is obtained; a first navigation route is generated based on the endurance information and the working farmland environment model; the working farmland environment model is divided into key weeding areas based on feature identifiers in the three-dimensional plant model, and a division result is generated; the first navigation route is adjusted based on the division result, and an adjusted route is generated; the position of the servo robot is adjusted based on the peripheral visual image, the robot position information, and the adjustment route, and the initial position and the end position are determined; a visual navigation route is generated based on the initial position, the end position, and the adjustment route.

[0098] In this embodiment, when developing a visual navigation route, it is necessary not only to ensure that the developed route is more accurate but also to better match the actual weed growth conditions and the servo robot's battery life, ensuring maximum weed removal efficiency while ensuring accurate movement. The servo robot's battery life information is collected, and a first navigation route is generated based on this battery life information and the working farmland environment model. The first navigation route is the optimal route for the servo robot, regardless of weed growth conditions. That is, under the first navigation route, the servo robot can complete the entire working farmland environment model in the shortest time and with the least battery life replenishment.

[0099] After generating the first navigation route, the area with lush weed growth in the working farmland environment model is determined based on the feature identifier, and the area is used as the key weeding area. In a working farmland environment model, multiple key weeding areas are allowed. After the division is completed, the division result is obtained. The first navigation route is adjusted based on the division result to obtain an adjusted route. When weeding, the servo robot needs to move along the adjusted route. After obtaining the adjusted route, the initial position of the servo robot is adjusted based on the peripheral visual image and robot position information collected by the servo robot, and the initial position and the end position are determined. That is, the forward direction of the servo robot is determined through the peripheral visual image, and the node in the adjustment route closest to the servo robot is determined based on the robot position information. The nearest node is used as the initial position. The end position can be directly obtained based on the initial position and the adjustment route. The forward direction of the servo robot is adjusted and moved to the initial position. The visual navigation route is obtained in combination with the end position.

[0100] Furthermore, the first navigation route is adjusted based on the division result, and the generation of the adjusted route includes: determining the regional proportion and regional position of the key weeding area based on the division result; judging whether the servo robot can process it in a single series based on the regional proportion and regional position; if the servo robot can process it in a single series, generating a key weeding route based on the regional position; adjusting the first navigation route based on the key weeding route, regional position and endurance information to generate an adjusted route; if the servo robot cannot process it in a single series, adjusting the first navigation route based on the regional proportion, regional position and endurance information to generate an adjusted route.

[0101] When adjusting the first navigation route, the regional proportion and regional position of the key areas must first be determined based on the division results. When weeding, the key weeding areas are processed first. If the regional positions are on a route, and the servo robot's endurance information is determined to be able to meet the weeding needs of all key weeding areas through the regional proportion, it will be determined that the servo robot can process it in a single series. The key weeding areas will be connected in series according to the regional positions to obtain the key weeding route. The first navigation route will be adjusted according to the key weeding route to obtain the adjusted route. If the regional positions are not on a processing route or the endurance information does not meet the weeding needs of all key weeding areas, it will be determined that the servo robot cannot process it in a single series. Non-key areas will be added according to the regional proportion and regional position combined with the endurance information to form an area that combines non-key areas and key weeding areas. It should be noted that in the obtained area, the key weeding area needs to occupy more than 80%, so that the first navigation route is adjusted to generate an adjusted route.

[0102] In this embodiment, the predicted position of the first weed is determined based on the initial position, the working farmland environment model and the visual navigation route; the visual image data and plant position data of the servo robot when it reaches the initial position are obtained; based on the initial position, the visual image data and the plant position data, it is determined whether the predicted position is accurate; if the predicted position is accurate, weeding navigation is performed based on the visual navigation route and the working farmland environment model; if the predicted position is inaccurate, the predicted positions of all plants are corrected based on the data image data and the plant position data to generate a corrected position; and weeding navigation is performed based on the corrected position visual navigation route and the working farmland environment model.

[0103] When the visual navigation route is constructed and weeding is performed according to the visual navigation route, there may be errors in the weeding position. Therefore, when moving to the position of the first weed for weeding, it is necessary to first confirm the weeding position. When an error is found, it is corrected in time to ensure the accuracy of weeding. Since a visual navigation route has been established, there is only an error in distance. This error may be caused by the initial position of the servo robot or the conversion between coordinates. When adjusting the distance error, the position of the first weed is first determined based on the initial position, the farmland environment model and the visual navigation route. This position is used as the predicted position, and the servo robot is controlled to move to the initial position. The servo robot uses multiple sensors set by itself to measure the position of the first weed at the initial position to obtain visual image data and plant position data. When the initial position remains consistent, the visual image data is combined to determine whether the plant position data is consistent with the initial position. If they are consistent, the predicted position is determined to be accurate, and weeding navigation can be directly performed according to the visual navigation route and the working farmland environment model. If they are inconsistent, the predicted position is determined to be inaccurate, and the plant position data will be used to correct the predicted positions of all plants in the working farmland environment model to obtain the corrected position. After the correction is completed, weeding navigation is performed according to the corrected position, the visual navigation route and the working farmland environment model.

[0104] In this embodiment, during the weeding process, there may be obstacles in the visual navigation route. In order to ensure the normal progress of weeding and the safety of the servo robot itself, visual information of the surroundings is collected in real time during the weeding process. Since the collected visual information is predictive, that is, when visual information is collected at point a, the environmental data of point b in front can be collected at the same time, so that the obstacle avoidance strategy can be formulated in advance, and the collected visual information is analyzed to determine whether there are abnormal objects in the visual information. If there are no abnormal objects, weeding will continue according to the visual navigation route. If there are abnormal objects, the volume of the abnormal objects will be calculated to determine whether the volume of the abnormal objects will cause forward obstruction. If it will not cause forward obstruction, weeding will continue according to the visual navigation route. If it will cause forward obstruction, further judgment will be required.

[0105] In the case of forward obstruction, the position of the abnormal object is calculated based on the visual information and the current position, and the current position, planting position and the servo robot's own external parameters are combined to determine whether the abnormal object can be bypassed. The bypass here is to make a small turn on the original straight path. Since the ridges of the corn field are long, straight and uninterrupted ridges, the servo robot cannot crush the ridges when bypassing, and can only make one side of the tires approach the ridges or move away from the ridges. If the bypass is possible, it will start to bypass when it reaches a preset distance from the abnormal object, and quickly return to the normal driving position after the bypass is completed. When bypassing, the offset distance of the bypass is also calculated, and the weeding position is adjusted based on the offset distance. If the bypass is not possible, a blocking signal will be generated and sent to the staff's mobile terminal to remind the staff to deal with the obstacle.

[0106] Let’s take an example based on the above scheme:

[0107] There is a corn farmland, and its panoramic image data and image ratio data are collected. According to the analysis of the panoramic image data, the regional image of the corn farmland is a square with a side length of 4. The planting position includes two ridges. The width of the two ridges is 0.5, the spacing between the two ridges is 1, and the length of the two ridges is 4. The image ratio is 1:100, that is, 1 cm in the image corresponds to 1 meter in reality. Afterwards, a farmland model with the same square shape is searched in the preset model database. At this time, the farmland model is an empty model without plants and ridges. After the farmland model is selected, the actual model size is determined according to the image ratio, the farmland model is adjusted according to the model size, and the planting position is added to the adjusted farmland model according to the model size to obtain a three-dimensional farmland model.

[0108] After constructing the farmland 3D model, the plant 3D model is constructed based on the obtained crop identification data and weed identification data. Figure 2 The crop identification data shown is constructed as Figure 3 The plant 3D model shown in the figure corresponds to the actual position of the plant and is rendered, so that the robot can quickly identify the crops that cannot be destroyed. The constructed plant 3D model is then resized and integrated into the farmland 3D model to generate the final working farmland environment model. Since there are more than one ridges in the working farmland environment model, the ridges are numbered when constructing the farmland 3D model. Then, the plant 3D model is added and integrated into the farmland 3D model according to the corresponding numbers to complete the construction of the working farmland environment model.

[0109] The visual navigation route is generated based on the constructed farmland 3D model. When generating the first navigation route, the endurance information and the working farmland environment model are used to automatically generate the first navigation route. For example:

[0110] "class AgriculturalNavigator:

[0111] def __init__(self, env_model, battery_range):

[0112] self.grid = self.process_env(env_model)

[0113] self.battery = battery_range * 1000

[0114] self.rows, self.cols = self.grid.shape

[0115] def process_env(self, model):

[0116] grid = np.zeros((50, 100))

[0117] for point in model:

[0118] x, y, _, t = point

[0119] grid[int(y)][int(x)] = 0 if t == 1 else 1

[0120] return grid

[0121] def heuristic(self, a, b):

[0122] return abs(a[0]-b[0]) + abs(a[1]-b[1])

[0123] def a_star(self, start, end):

[0124] open_set = []

[0125] heapq.heappush(open_set, (0, start))

[0126] came_from = {}

[0127] g_score = defaultdict(lambda: float('inf'))

[0128] g_score[start] = 0

[0129] while open_set:

[0130] current = heapq.heappop(open_set)[1]

[0131] if current == end:

[0132] return self.reconstruct_path(came_from, current)

[0133] for dx, dy in [(-1,0), (1,0), (0,1), (0,-1)]:

[0134] neighbor = (current[0]+dx, current[1]+dy)

[0135] if 0 <= neighbor[0] < self.rows and 0 <= neighbor[1]< self.cols:

[0136] if self.grid[neighbor] == 1:

[0137] continue

[0138] tentative_g = g_score[current] + 1

[0139] if tentative_g < g_score[neighbor] and tentative_g < self.battery:

[0140] came_from[neighbor] = current

[0141] g_score[neighbor] = tentative_g

[0142] f = tentative_g + self.heuristic(neighbor, end)

[0143] heapq.heappush(open_set, (f, neighbor))

[0144] return None

[0145] def reconstruct_path(self, came_from, current):

[0146] path = []

[0147] while current in came_from:

[0148] path.append(current)

[0149] current = came_from[current]

[0150] return path[::-1]".

[0151] After inputting the endurance information and the working farmland environment model, the first navigation route can be directly obtained. After generating the first navigation route, the key weeding areas are divided. If there are key weeding areas, the first navigation route will be further adjusted, for example:

[0152] "class RouteOptimizer(AgriculturalNavigator):

[0153] def __init__(self, env_model, battery_range):

[0154] super().__init__(env_model, battery_range)

[0155] self.area_graph = {}

[0156] def calculate_coverage(self, path):

[0157] return len(path) * 1.0

[0158] def can_single_handle(self, main_path, areas):

[0159] main_len = self.calculate_coverage(main_path)

[0160] area_paths = [self.a_star(a['entry'], a['exit']) for a inareas]

[0161] total_area_len = sum(self.calculate_coverage(p) for p inarea_paths)

[0162] return (main_len + total_area_len) <= self.battery

[0163] def generate_weeding_route(self, areas):

[0164] route = [areas[0]['entry']]

[0165] unvisited = [a['entry'] for a in areas[1:]]

[0166] while unvisited:

[0167] nearest = min(unvisited, key=lambda x: self.heuristic(route[-1], x))

[0168] route.append(nearest)

[0169] unvisited.remove(nearest)

[0170] return route

[0171] def adjust_route(self, main_path, areas):

[0172] if self.can_single_handle(main_path, areas):

[0173] weeding_path = self.generate_weeding_route(areas)

[0174] merged = self.merge_routes(main_path, weeding_path)

[0175] return merged[:self.battery]

[0176] else:

[0177] return self.split_and_adjust(main_path, areas)

[0178] def merge_routes(self, main, sub):

[0179] connect_point = min(sub[0], key=lambda x: self.heuristic(x, main[-1]))

[0180] return main + self.a_star(main[-1], connect_point) + sub

[0181] def split_and_adjust(self, main_path, areas):

[0182] adjusted = []

[0183] current_len = 0

[0184] for segment in main_path:

[0185] if current_len + 1 > self.battery:

[0186] break

[0187] adjusted.append(segment)

[0188] current_len += 1

[0189] for area in areas.copy():

[0190] path = self.a_star(segment, area['entry'])

[0191] if path and (current_len + len(path)) <= self.battery:

[0192] adjusted += path

[0193] current_len += len(path)

[0194] areas.remove(area)

[0195] return adjusted”.

[0196] The first navigation route is adjusted to generate an adjusted route, and the adjusted route is used for weeding. It should be noted that the generation of the first navigation path and the adjusted route includes but is not limited to the above-mentioned example running program, and the language and running logic used by the running program can also be adjusted according to actual needs, which is not specifically limited here.

[0197] Figure 4 A structural block diagram of a visual navigation system 200 for a servo robot provided in an embodiment of the application.

[0198] like Figure 4 As shown, the visual navigation system 200 for the servo robot mainly includes:

[0199] The farmland data acquisition module 201 is used to acquire panoramic image data and image scale data of the target farmland;

[0200] The farmland model construction module 202 is used to construct a three-dimensional farmland model of the target farmland based on the panoramic image data and the image scale data;

[0201] The crop information acquisition module 203 is used to obtain crop identification data, weed identification data, the peripheral visual image of the servo robot, and the robot position information of the target farmland;

[0202] a plant model building module 204 for generating a three-dimensional plant model based on the crop identification data and the weed identification data;

[0203] An environment model building module 205 is used to build a working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model;

[0204] The navigation route construction module 206 is used to construct a visual navigation route for the servo robot based on the surrounding visual image, the robot position information and the working farmland environment model.

[0205] As an optional implementation of this embodiment, the farmland model construction module 202 is specifically used to determine the regional image and planting position of the target farmland based on the panoramic picture data; match the regional image with the farmland model in the preset model database to determine the regional similarity between the regional image and the farmland model; judge whether the farmland model needs to be morphologically adjusted based on the regional similarity; if the farmland model needs to be morphologically adjusted, determine the regional image to adjust the farmland model to generate an adjusted farmland model; construct a three-dimensional farmland model based on the image scale data, the adjusted farmland model and the planting position; if the farmland model does not need to be morphologically adjusted, construct a three-dimensional farmland model based on the image scale data, the farmland model and the planting position.

[0206] As an optional implementation of this embodiment, the plant model construction module 204 is specifically used to determine species morphological difference information based on crop identification data and weed identification data; determine crop positions based on the crop identification data, and determine weed positions based on the weed identification data; add the crop identification data and weed identification data to the planting positions according to the crop positions and weed positions to generate a plant model; and add feature identification to the plant model based on the species morphological difference information and preset identification rules to generate a three-dimensional plant model.

[0207] As an optional implementation of this embodiment, the environmental model construction module 205 is specifically used to obtain plant size data of the plant three-dimensional model; calculate the farmland size data of the farmland three-dimensional model based on the panoramic picture data and image ratio data; calculate the position size of the planting position based on the farmland size data; calculate the size ratio of the plant three-dimensional model based on the position size and the plant size data; adjust the plant three-dimensional model based on the size ratio to generate an adjusted plant model; merge the adjusted plant model with the farmland three-dimensional model according to the planting position to construct a working farmland environment model.

[0208] As an optional implementation of this embodiment, the navigation route construction module 206 includes:

[0209] Endurance information acquisition module, used to obtain the endurance information of the servo robot;

[0210] A first route generating module, configured to generate a first navigation route based on the endurance information and the working farmland environment model;

[0211] A division result generation module is used to divide the working farmland environment model into key weeding areas based on feature identifiers in the three-dimensional plant model and generate division results;

[0212] an adjusted route generating module, configured to adjust the first navigation route based on the division result and generate an adjusted route;

[0213] A position adjustment determination module is used to adjust the position of the servo robot based on the peripheral visual image, the robot position information, and the adjustment route, and determine the initial position and the end position;

[0214] The visual route generation module is used to generate a visual navigation route based on the initial position, the end position and the adjusted route.

[0215] In this optional embodiment, the adjustment route generation module is specifically used to determine the regional proportion and regional position of the key weeding area based on the division results; judge whether the servo robot can process it in a single series based on the regional proportion and regional position; if the servo robot can process it in a single series, generate a key weeding route based on the regional position; adjust the first navigation route based on the key weeding route, regional position and endurance information to generate an adjusted route; if the servo robot cannot process it in a single series, adjust the first navigation route based on the regional proportion, regional position and endurance information to generate an adjusted route.

[0216] As an optional implementation of this embodiment, the visual navigation system 200 for the servo robot further includes:

[0217] A predicted position determination module is used to determine the predicted position of the first weed based on the initial position, the working farmland environment model and the visual navigation route;

[0218] Related data acquisition module, used to obtain visual image data and plant position data when the servo robot reaches the initial position;

[0219] A position accuracy judgment module is used to judge whether the predicted position is accurate based on the initial position, visual image data and plant position data;

[0220] A first weeding navigation module is used for weeding navigation based on a visual navigation route and a working farmland environment model;

[0221] A corrected position generation module is used to correct the predicted positions of all plants based on the image data and the plant position data to generate corrected positions;

[0222] The second weeding navigation module is used to perform weeding navigation based on the corrected position visual navigation route and the working farmland environment model.

[0223] In one example, the module in any of the above devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0224] For another example, when the modules in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0225] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0226] Figure 5 This is a structural block diagram of the electronic device 300 provided in an embodiment of the present application.

[0227] like Figure 5 As shown, the electronic device 300 includes a processor 301 and a memory 302 , and may further include an information input / information output (I / O) interface 303 , one or more communication components 304 , and a communication bus 305 .

[0228] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps of the above-mentioned visual navigation method for a servo robot. The memory 302 is used to store various types of data to support the operation of the electronic device 300. For example, these data may include instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0229] The I / O interface 303 provides an interface between the processor 301 and other interface modules, which may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 304 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more thereof, therefore, the corresponding communication component 304 may include: Wi-Fi components, Bluetooth components, NFC components.

[0230] The electronic device 300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the visual navigation method for a servo robot given in the above embodiment.

[0231] Communication bus 305 may include a path for transmitting information between the aforementioned components. Communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Communication bus 305 may be divided into an address bus, a data bus, a control bus, and the like.

[0232] The electronic device 300 may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc., and may also be servers, etc.

[0233] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned visual navigation method for a servo robot are implemented.

[0234] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0235] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0236] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of application involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the aforementioned application concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions applied for in this application.

Claims

1. A visual navigation method for a servo robot, characterized in that: include: Obtain panoramic image data and image scale data of the target farmland; constructing a three-dimensional farmland model of the target farmland based on the panoramic picture data and the image scale data; Acquiring crop identification data, weed identification data, a peripheral visual image of the servo robot, and robot position information of the target farmland; generating a three-dimensional plant model based on the crop identification data and the weed identification data; Constructing a working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model; A visual navigation route of the servo robot is constructed based on the peripheral visual image, the robot position information and the working farmland environment model.

2. The method according to claim 1, characterized in that The constructing of the three-dimensional farmland model of the target farmland based on the panoramic picture data and the image scale data includes: Determining the regional image and planting position of the target farmland based on the panoramic image data; Matching the regional image with a farmland model in a preset model database to determine the regional similarity between the regional image and the farmland model; Determining whether the farmland model needs to be morphologically adjusted based on the regional similarity; If the farmland model needs to be morphologically adjusted, determining the regional image to adjust the farmland model to generate an adjusted farmland model; constructing a three-dimensional farmland model based on the image scale data, the adjusted farmland model, and the planting positions; If the farmland model does not need to be morphologically adjusted, a three-dimensional farmland model is constructed based on the image scale data, the farmland model and the planting positions.

3. The method according to claim 2, characterized in that Generating a three-dimensional plant model based on the crop identification data and the weed identification data includes: determining species morphological difference information based on the crop identification data and the weed identification data; determining crop locations based on the crop identification data, and determining weed locations based on the weed identification data; adding the crop identification data and the weed identification data to the planting location according to the crop location and the weed location to generate a plant model; Based on the species morphological difference information and preset identification rules, feature identification is added to the plant model to generate a three-dimensional plant model.

4. The method according to claim 2, characterized in that The constructing of the working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model includes: Acquiring plant size data of the plant three-dimensional model; Calculating farmland size data of the three-dimensional farmland model based on the panoramic picture data and the image scale data; Calculating the location size of the planting location based on the farmland size data; Calculating a size ratio of the three-dimensional plant model based on the position size and the plant size data; Adjusting the plant three-dimensional model based on the size ratio to generate an adjusted plant model; The adjusted plant model is fused with the farmland three-dimensional model according to the planting position to construct a working farmland environment model.

5. The method according to claim 3, characterized in that The step of constructing the visual navigation route of the servo robot based on the peripheral visual image, the robot position information, and the working farmland environment model includes: Obtaining the endurance information of the servo robot; generating a first navigation route based on the endurance information and the working farmland environment model; Dividing the working farmland environment model into key weeding areas based on feature identifiers in the three-dimensional plant model, and generating a division result; adjusting the first navigation route based on the division result to generate an adjusted route; Adjusting the position of the servo robot based on the peripheral visual image, the robot position information, and the adjustment route to determine an initial position and an end position; A visual navigation route is generated based on the initial position, the end position, and the adjusted route.

6. The method according to claim 5, characterized in that The adjusting the first navigation route based on the division result to generate the adjusted route includes: Determine the regional proportion and regional location of the key weed control areas based on the division results; Determining whether the servo robot is capable of single serial processing based on the area proportion and the area position; If the servo robot is capable of single serial processing, a focused weeding route is generated based on the location of the area; Adjusting the first navigation route based on the key weeding route, the regional location, and the endurance information to generate an adjusted route; If the servo robot cannot process the process in a single series, the first navigation route is adjusted based on the area proportion, the area position and the endurance information to generate an adjusted route.

7. The method according to claim 5, characterized in that After constructing the visual navigation route of the servo robot based on the peripheral visual image, the robot position information and the working farmland environment model, the method further includes: determining a predicted position of a first weed based on the initial position, the working farmland environment model, and the visual navigation route; acquiring visual image data and plant position data when the servo robot reaches the initial position; determining whether the predicted position is accurate based on the initial position, the visual image data, and the plant position data; If the predicted position is accurate, weeding navigation is performed based on the visual navigation route and the working farmland environment model; If the predicted position is inaccurate, the predicted positions of all plants are corrected based on the image data and the plant position data to generate a corrected position; Weeding navigation is performed based on the corrected position, the visual navigation route, and the working farmland environment model.

8. A visual navigation system for a servo robot, characterized in that: include: The farmland data acquisition module is used to obtain panoramic image data and image scale data of the target farmland; A farmland model construction module, configured to construct a three-dimensional farmland model of the target farmland based on the panoramic image data and the image scale data; a crop information acquisition module, configured to acquire crop identification data, weed identification data, a peripheral visual image of the servo robot, and robot position information of the target farmland; a plant model construction module, configured to generate a three-dimensional plant model based on the crop identification data and the weed identification data; An environment model construction module, configured to construct a working farmland environment model based on the farmland three-dimensional model and the plant three-dimensional model; A navigation route construction module is used to construct a visual navigation route for the servo robot based on the peripheral visual image, the robot position information and the working farmland environment model.

9. An electronic device, characterized in that: comprising a processor coupled to a memory; The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The method comprises a computer program or an instruction, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images