Method and device for identifying and administering red fire ant nests based on robot dogs

Abstract

The embodiment of the present invention relates to the technical field of red fire ant nest identification, and discloses a method and device for identifying and administering red fire ant nests based on a robot dog. The method comprises: taking photos by a robot dog to obtain multiple images, and associating each image with corresponding position information; converting the image into a three-dimensional scene graph; inputting the three-dimensional scene graph into a pre-trained YOLOX model for identification, and determining a target image, which is an image corresponding to the position of a red fire ant nest; setting a medication path based on the target image and its position information, and administering medication to the red fire ant nest corresponding to the target image by a robot dog. The implementation of the embodiment of the present invention can improve the recognition efficiency, obtain accurate positioning, reduce personnel input, and improve the efficiency of ant killing, which has a positive effect on the prevention and control of red fire ant invasion.

Description

Technical Field

[0001] The present invention relates to the technical field of red fire ant nest identification, and in particular to a method and device for red fire ant nest identification and drug administration based on a robot dog. Background Art

[0002] Red fire ants are native to Brazil, Argentina and other countries in South America. Over the past 90 years, the species has expanded from its place of origin to countries and regions such as North America, Australia and Asia, causing great harm to agricultural and forestry production, human and animal health, biodiversity and public safety.

[0003] The invasion of red fire ants has caused crop damage, abandoned farmland, biting of poultry, stinging of the public, and endangerment to the lives of sensitive groups in some parts of southern my country, posing a potential threat to public safety, agricultural and forestry production, and ecosystem security.

[0004] In the early days, the U.S. Department of Agriculture used organochlorine chemicals similar to DDT to control it, but the lethal chemicals often only killed worker ants foraging outside the nest and could not effectively kill the queen ants in the nest. Since the 1990s, the United States has used baits (spread by airplanes) to kill ants on a large scale, which has achieved significant results, and about 90% of ant nests can be eliminated that year. However, the number of red fire ants usually exceeds the original scale within three years. Analysis shows that baits will also kill a large number of native ant species, and the surviving red fire ants can always make a comeback and occupy the living space of the original native ants. It can be seen from the failed experience of large-scale use of chemicals in the United States that it is necessary to find accurate drug delivery to control and eliminate red fire ants.

[0005] Our country’s current prevention and control strategy is hotspot control. When someone discovers a nest and reports it, local epidemic prevention personnel will go and apply bait. However, currently the bait can only prevent ~85% of the invading red fire ant colonies. The main reason may be that the bait is a mixture of chemical drugs and degreased corn granules and soybean oil. Once it is not fresh or damp, it will reduce its attractiveness to ants. Moreover, contact-killing agents may harm other insects and are only recommended for small-scale use, so the effect is limited. In addition, there are physical methods such as injecting low-temperature liquid nitrogen or hot steam to kill them, but these methods require a lot of human resources and cannot be used on a large scale. Summary of the invention

[0006] In response to the above-mentioned defects, the embodiments of the present invention disclose a method and device for identifying and administering red fire ant nests based on a robot dog. The method implants relevant models into the robot dog for real-time identification and precise point administration, thereby completing intelligent identification and ant extermination. This can not only improve identification efficiency and obtain precise positioning, but also reduce manpower input, and at the same time improve ant extermination efficiency, thus playing a positive role in the prevention and control of the invasion of red fire ants.

[0007] The first aspect of the embodiment of the present invention discloses a method for identifying and administering red fire ant nests based on a robot dog, the method comprising:

[0008] Taking photos with the robot dog to obtain multiple images, and associating each image with corresponding location information;

[0009] converting the image into a three-dimensional scene graph;

[0010] The three-dimensional scene graph is input into a pre-trained YOLOX model for recognition to determine a target image, wherein the target image is an image corresponding to the location of a red fire ant nest;

[0011] A medication route is set based on the target image and its position information, and the robot dog is used to administer medication to the red fire ant nest at the corresponding position of the target image.

[0012] As an optional implementation, in the first aspect of the embodiment of the present invention, the pre-trained YOLOX model includes:

[0013] Collect sample images by web crawling and / or field photography to construct a sample image set, wherein the sample images include a plurality of images of red fire ant nests and a plurality of images of native ant nests;

[0014] Using the acquired sample image set, the sample images are labeled and classified. The labeled files contain the category and location coordinate information of the ant nest.

[0015] The nest target features are extracted from the sample image set according to the annotation file, and a single-camera binocular stereo vision algorithm is used to construct a three-dimensional model of the nest based on the extracted multi-angle nest features;

[0016] A complete three-dimensional scene graph is constructed based on the three-dimensional model of the nest and the scene where the nest is located, and all scene graphs are aggregated to form a three-dimensional model dataset of the ant nest;

[0017] The three-dimensional model dataset is constructed into a COCO input dataset of the YOLOX model standard by using model rendering and coordinate transformation to train the YOLOX model.

[0018] As an optional implementation manner, in the first aspect of the embodiment of the present invention, after constructing the sample image set, the method further includes:

[0019] The sample images are further processed using manual labeling and classification methods using the Labelme program. The processed images and labels are converted from the original images and label data into a standard data set of the YOLOX model through the labelme2coco program. The constructed sample image set is divided into training data and verification data of the YOLOX model through segmentation.

[0020] As an optional implementation, in the first aspect of the embodiment of the present invention, a complete three-dimensional scene graph is constructed based on the three-dimensional model of the nest and the scene in which the nest is located, including:

[0021] Step 11: Acquire a texture-rich environment background on site to ensure that the object and the background are relatively still during the picture shooting process;

[0022] Step 12: obtain camera data by calibrating the camera, and use the above data to determine the correspondence between the feature points of the sample image and the points in the world coordinate system. The camera data includes focal length, principal point, rotation angle and translation of the camera;

[0023] Step 13, matching and extracting feature points of the images, matching the feature points of sample images of the same detection target collected at different angles in pairs, screening and extracting correctly matched feature point pairs, using these feature point pairs to arrange and combine images of the same target at different angles, so that the arrangement and combination of these images show the position coordinates of various parts of the original detection target in three-dimensional space, and finally extracting the appropriate position coordinates under the images at different angles as the spatial positions of various parts of the original detection target;

[0024] Step 14: According to the spatial position coordinate information of different parts of the original detection target obtained in step 13, the required foreground, i.e., the spatial parts of the original detection target, are segmented from the image set with a certain arrangement and combination obtained in step 13 by using manual annotation or image segmentation technology, and the edge contour is extracted at the same time, and all points within and on the contour line are retained, and the remaining points are eliminated to obtain the spatial set points surrounding the object;

[0025] Step 15: Use the Delaunay triangulation algorithm to fit the surface of the target object using a triangular plane, project the spatial point set obtained in step 14 onto a certain plane, segment the point set on the plane, and restore the segmented triangulated network to space to achieve a triangulated model of the object, that is, a three-dimensional object model;

[0026] Step 16: Use the same method to obtain a scene graph model, and combine the three-dimensional object model with the scene graph model to form a complete three-dimensional scene graph containing the object model to be identified.

[0027] As an optional implementation, in the first aspect of the embodiment of the present invention, the three-dimensional model dataset is constructed using model rendering and coordinate transformation to construct a COCO input dataset of the YOLOX model standard, and the YOLOX model is trained, including:

[0028] Step 21, construct a model rendering function: R (Mo, Mb, po, eo, to, pw, ew, hi, wi, L); wherein R is a scene rendering function, Mo is a 3D model of the object to be measured, Mb is a scene background model, po, eo, to are respectively the center point coordinates, Euler angles, and texture parameters of Mo in M, pw, ew are respectively the center point coordinates and Euler angles of the image window in M; hi, wi are respectively the height and width of the output image, and L is the lighting condition of the three-dimensional scene graph;

[0029] Step 22: The three-dimensional scene graph is passed through a rendering function multiple times to obtain an original data set, and the coordinates of the upper left vertex, the lower right vertex, and the width and height of the image of the bounding box are calculated to obtain the coordinates of the center point, width, height, and relative ratio of the image width and height of the bounding box based on the coordinates of the upper left vertex, the coordinates of the lower right vertex, and the width and height of the image of the bounding box in the marked center of the original data set;

[0030] Step 23: Convert the three-dimensional scene graph into the YOLOX standard COCO dataset using the relative proportion obtained in step 22.

[0031] As an optional implementation, in the first aspect of the embodiment of the present invention, the three-dimensional model dataset is constructed using model rendering and coordinate transformation to construct a COCO input dataset of the YOLOX model standard, and the YOLOX model is trained, including:

[0032] Step 31: Input the red fire ant nest model dataset of different sizes processed into COCO format into the YOLOX model;

[0033] Step 32: Pass the input data through the Backbone feature extraction network. The output of this layer of network is three effective feature layers of different sizes obtained by scaling the original image model accordingly. Compared with the original image model size H*W, the outputs of the three feature layers are H / 8*W / 8, H / 16*W / 16, and H / 32*W / 32, respectively, where H and W are the height and width of the image, respectively.

[0034] Step 33: Pass the three feature layers in step 32 through the FPN layer and the PAN layer respectively to obtain outputs of three original feature map sizes, and then use the outputs as inputs of the Decouple Head layer;

[0035] Step 34: In the Decouple Head layer, the feature map is separated, classified and located in the form of a decoupled head, and finally the target category, location information and IoU in the feature map are obtained through classification and regression tasks;

[0036] Step 35: In order to facilitate the remote and accurate training of the robot dog's target recognition model, a training website is built to achieve communication with the robot dog and operate the training of its model in real time.

[0037] As an optional implementation, in the first aspect of the embodiment of the present invention, communication with the robot dog is achieved by building a training website, including:

[0038] Step 41: A lightweight website built based on the third-party python library Flask is used to build a server through the intranet penetration service and deploy the website to the public network;

[0039] Step 42: Deploy the YOLOX model to the website to facilitate remote model training and visualization hyperparameter adjustment;

[0040] Step 43: Set up a training part and a prediction part according to the division of website functions. The training part is used by technicians to evaluate the effects of different models, and the prediction part is used to detect the recognition status of different models so as to select the required model for deployment according to the actual situation.

[0041] Step 44: Based on the feature that the website can use IP / TCP technology to establish real-time communication with the robot dog, the images sent back by the website are used to evaluate the field detection effect of the robot dog and provide correction feedback;

[0042] Step 45: Based on the feedback results and combined with GPS positioning technology, a suitable exploration path is planned for the robot dog in real time, and the robot dog is remotely controlled to interact with the suspected ant nest target.

[0043] As an optional implementation, in the first aspect of the embodiment of the present invention, a medication delivery path is set based on a target image and its position information, and medication is delivered to a red fire ant nest at a position corresponding to the target image by the robot dog, including:

[0044] Step 51: The robot dog GPS patrols to detect the occurrence of red fire ant disasters, which includes the following steps:

[0045] Step 511: By arranging the path on the map, the robot dog automatically screens the nests of red fire ants according to the screening path;

[0046] Step 512: Mark the existing GPS location of the red fire ant nest, and have the robot dog continuously monitor the disinfecting effect of the ant nest at different time intervals;

[0047] Step 513, automatically statistically analyzing the density of red fire ant nests within a unit area, and observing the migration and mating flight of ant colonies;

[0048] Step 52: The robot dog knocks the target ant nest with its front legs to dynamically identify whether forage ants and guard ants emerge from the ant nest, accurately determine whether the ant nest is an active red fire ant nest, and reduce the recognition error rate to zero;

[0049] Step 53: The robot dog uses two rounds of spraying. In the first round, borax is used to seal the red fire ants in the nest and start disinfecting them. In the second round, orange oil, which is environmentally friendly, is repeatedly sprayed around the nest.

[0050] The second aspect of the embodiment of the present invention discloses a device for identifying and administering red fire ant nests based on a robot dog, which includes:

[0051] A shooting unit, used to take pictures with the robot dog, obtain multiple images, and associate each image with corresponding position information;

[0052] A conversion unit, used for converting the image into a three-dimensional scene graph;

[0053] A recognition unit, used for inputting the three-dimensional scene graph into a pre-trained YOLOX model for recognition, and determining a target image, wherein the target image is an image corresponding to the location of a red fire ant nest;

[0054] The drug delivery unit is used to set a drug delivery path based on the target image and its position information, and to deliver drugs to the red fire ant nest at the corresponding position of the target image through the robot dog.

[0055] The third aspect of an embodiment of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute a red fire ant nest identification and medication method based on a robot dog disclosed in the first aspect of an embodiment of the present invention.

[0056] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute a robot dog-based red fire ant nest identification and medication method disclosed in the first aspect of an embodiment of the present invention.

[0057] The fifth aspect of the embodiments of the present invention discloses a computer program product. When the computer program product is run on a computer, the computer is enabled to execute a method for identifying and administering red fire ant nests based on a robot dog disclosed in the first aspect of the embodiments of the present invention.

[0058] The sixth aspect of an embodiment of the present invention discloses an application publishing platform, which is used to publish a computer program product. When the computer program product runs on a computer, the computer executes a red fire ant nest identification and medication method based on a robot dog disclosed in the first aspect of an embodiment of the present invention.

[0059] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0060] The embodiment of the present invention adopts a robot dog instead of manual labor to obtain the image of the red fire ant nest, thereby realizing the real-time identification, positioning and elimination of the red fire ant nest, improving the detection efficiency, reducing the input of personnel, and preventing the red fire ants from biting people; secondly, the use of the robot dog for real-time identification and multiple rounds of targeted drug administration to eliminate ants not only protects other species and the ecological environment, but also achieves the effect of eliminating ants, while greatly reducing the waste of manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 It is a flow chart of a method for identifying and administering red fire ant nests based on a robot dog according to an embodiment of the present invention;

[0063] Figure 2 It is a schematic diagram of the process of pre-training the YOLOX model disclosed in an embodiment of the present invention;

[0064] Figure 3 It is a schematic diagram of a process of constructing a three-dimensional scene graph disclosed in an embodiment of the present invention;

[0065] Figure 4 are some three-dimensional scene graphs containing models of objects to be identified disclosed in the embodiments of the present invention;

[0066] Figure 5 It is a schematic diagram of the process of constructing a COCO input data set of the YOLOX model standard disclosed in an embodiment of the present invention;

[0067] Figure 6 is a schematic diagram of a sub-process for training the YOLOX model disclosed in an embodiment of the present invention;

[0068] Figure 7 The embodiment of the present invention discloses Figure 4 The identification result of the location of the red fire ant nest;

[0069] Figure 8 It is a schematic diagram of the process of precise drug delivery by a robot dog disclosed in an embodiment of the present invention;

[0070] Fig. 9 It is a structural schematic diagram of a device for identifying and administering red imported fire ant nests based on a robot dog disclosed in an embodiment of the present invention;

[0071] Fig.10 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] It should be noted that the terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" in the embodiments of the present invention and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0074] The embodiment of the present invention discloses a method and device for identifying and administering red fire ant nests based on a robot dog, which utilizes deep learning, automatic patrolling, precise spraying and other technologies to accurately identify and administer red fire ant nests in an area with a robot dog instead of manual methods. Among them, the deep learning method mainly uses a YOLOX neural network algorithm (YOLOX model) framework to perform image recognition on red fire ant nests. The training data used by the target detection algorithm comes from images of red fire ants collected from the Internet and images of red fire ant nests taken in the wild, and a single-camera binocular stereo vision algorithm is used to extract multi-angle image features of a nest, thereby constructing a three-dimensional model of the nest, and then combining the scene information of the ant nest to jointly construct a complete recognition image and render the scene graph, and finally transform the coordinates marked on the nest model to construct a standard data set (COCO-type image data set) of the YOLOX neural network. Through iterative training and testing of the data set, the high-precision training model is finally used as the core prediction model for the robot dog to detect red fire ant nests. The red fire ant nests in the wild are accurately patrolled by GPS path planning, and the red fire ants are sensitive to vibrations. The interaction between the robot dog and the nest is used to accurately determine the nest type. In the process of prevention and control, the technicians establish real-time communication with the robot dog, and accurately correct the errors of the returned images. At the same time, they mark the key areas where the red fire ant nests are densely populated, and use statistical principles to plan the distribution of the red fire ant nests. While improving the real-time recognition accuracy, it reduces costs and reduces the impact on the ecological environment. The following is a detailed description with the attached drawings.

[0075] Embodiment 1

[0076] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for identifying and administering red fire ant nests based on a robot dog disclosed in an embodiment of the present invention. Figure 1 As shown, the method for identifying and administering red fire ant nests based on a robot dog includes the following steps:

[0077] S110, taking photos with the robot dog to obtain multiple images, and associating each image with corresponding location information.

[0078] In a preferred embodiment of the present invention, the YOLOX model is directly arranged in the robot dog, so it can be directly determined whether the location of the image obtained by this photo shooting is a red fire ant nest. If so, the location information of the current location is recorded according to the built-in GPS module of the robot dog, thereby forming multiple target locations marked with red fire ant nests. Step S140 determines the medication path based on the target location to complete the medication operation.

[0079] Of course, in some other embodiments, the robot dog may send the image and location information to a remote processing device after taking a photo, and the processing device may recognize the target image based on the YOLOX model, that is, the image with the location of the red fire ant nest.

[0080] S120: Convert the image into a three-dimensional scene graph.

[0081] The method of converting an image into a three-dimensional scene graph is basically the same as the method of converting a sample image in training a YOLOX model into a three-dimensional scene graph, and reference may be made to the description of steps S133-S134.

[0082] S130, inputting the three-dimensional scene graph into a pre-trained YOLOX model for recognition, and determining a target image, wherein the target image is an image corresponding to the location of a red fire ant nest.

[0083] To pre-train the YOLOX model, please refer to Figure 2 As shown, it may include the following steps:

[0084] S131, collecting sample images by web crawling and / or field photography to construct a sample image set, wherein the sample images include a plurality of images of red fire ant nests and a plurality of images of native ant nests;

[0085] The sample images are further processed using manual labeling and classification methods using the Labelme program. The processed images and labels are converted from the original images and label data into a standard data set of the YOLOX model through the labelme2coco program. The constructed sample image set is divided into training data and verification data of the YOLOX model through segmentation.

[0086] Before training, a website is designed to realize remote control of model training, and a visual parameter setting interface and training result observation interface are designed on the training page to reduce the complexity of repeated parameter adjustment during the training process; the processed training data and verification data are input into the neural network model for training, and the optimal model is selected for the deployment of detection tasks by comparing the training effects of different models.

[0087] The optimal model is deployed to the onboard computer of the robot dog, and motion functions such as automatic obstacle avoidance in the wild are set for the robot dog. Then, the sensitivity of the robot dog is optimized and adjusted in a special test site.

[0088] S132, using the acquired sample image set, the sample images are labeled and classified, and the labeled files contain the category and location coordinate information of the ant nest;

[0089] S133, extracting nest target features from the sample image set according to the annotation file, and constructing a three-dimensional model of the nest using a single-camera binocular stereo vision algorithm according to the extracted multi-angle nest features;

[0090] S134, constructing a complete three-dimensional scene graph based on the three-dimensional model of the nest and the scene where the nest is located, and aggregating all the scene graphs to form a three-dimensional model data set of the ant nest.

[0091] Please refer to Figure 3 As shown, a complete three-dimensional scene graph is constructed based on the three-dimensional model of the nest and the scene where the nest is located, including:

[0092] S1341. Acquire a texture-rich environment background on site to ensure that the object and the background are relatively still during the picture shooting process;

[0093] S1342, using a camera calibration method to obtain camera data, and using the above data to determine the correspondence between feature points of the sample image and points in the world coordinate system, the camera data including focal length, principal point, rotation angle and translation amount of the camera;

[0094] S1343, matching and extracting feature points of the images, matching the feature points of sample images of the same detection target collected at different angles in pairs, screening and extracting correctly matched feature point pairs, using these feature point pairs to arrange and combine images of the same target at different angles, so that the arrangement and combination of these images show the position coordinates of various parts of the original detection target in three-dimensional space, and finally extracting suitable position coordinates under images at different angles as the spatial positions of various parts of the original detection target;

[0095] S1344, according to the spatial position coordinate information of different parts of the original detection target obtained in S1343, use manual marking or image segmentation technology to segment the required foreground, i.e., the spatial parts of the original detection target from the image set with a certain arrangement and combination method obtained in S1343, and extract its edge contour at the same time, retain all points within and on the contour line, and eliminate the remaining points to obtain the spatial set points surrounding the object;

[0096] S1345, using the Delaunay triangulation algorithm, using a triangular plane to fit the surface of the target object, projecting the spatial point set obtained in S1344 onto a certain plane, segmenting the point set on the plane, and restoring the segmented triangulated network to space to realize a triangulated model of the object, that is, a three-dimensional object model;

[0097] S1346. Use the same method to obtain a scene graph model, and combine the three-dimensional object model with the scene graph model to form a complete three-dimensional scene graph containing the object model to be identified.

[0098] Figure 4 Some three-dimensional scene graphs containing models of objects to be recognized are shown.

[0099] S135. Construct a COCO input data set of the YOLOX model standard by using model rendering and coordinate transformation on the three-dimensional model data set to train the YOLOX model.

[0100] Please refer to Figure 5 As shown, the model rendering and coordinate transformation are used to construct the COCO input dataset of the YOLOX model standard, including the following steps:

[0101] S1351. Construct a model rendering function: R (Mo, Mb, po, eo, to, pw, ew, hi, wi, L); wherein R is a scene rendering function, Mo is a 3D model of the object to be measured, Mb is a scene background model, po, eo, to are respectively the center point coordinates, Euler angles, and texture parameters of Mo in M, pw, ew are respectively the center point coordinates and Euler angles of the image window in M; hi, wi are respectively the height and width of the output image, and L is the lighting condition of the three-dimensional scene graph;

[0102] S1352, the three-dimensional scene graph is repeatedly passed through a rendering function to obtain an original data set, and the coordinates of the upper left vertex, the lower right vertex, and the width and height of the image of the bounding box are calculated to obtain the coordinates of the center point, width, height, and relative ratio of the image width and height of the bounding box based on the coordinates of the upper left vertex, the coordinates of the lower right vertex, and the width and height of the image of the bounding box of the marked center of the original data set;

[0103] S1353. Convert the three-dimensional scene graph into a YOLOX standard COCO dataset using the relative proportion obtained in S1352.

[0104] Please refer to Figure 6 As shown, the YOLOX model is trained, including:

[0105] S1354, inputting the red fire ant nest model dataset of different sizes processed into COCO format into the YOLOX model;

[0106] S1355, the input data is passed through the Backbone feature extraction network. The output of this layer of network is three effective feature layers of different sizes obtained by scaling the original image model accordingly. Compared with the size H*W of the original image model, the outputs of the three feature layers are H / 8*W / 8, H / 16*W / 16, and H / 32*W / 32, respectively, where H and W are the height and width of the image, respectively.

[0107] S1356, passing the three feature layers in step S1352 through the FPN layer and the PAN layer respectively to obtain outputs of three original feature map sizes, and then using the outputs as inputs of the Decouple Head layer;

[0108] S1357, in the Decouple Head layer, the feature map is separated, classified and located in the form of a decoupled head, and finally the target category, location information and IoU in the feature map are obtained through classification and regression tasks;

[0109] Figure 7 Shows the Figure 4 The identification result of the red fire ant nest in the figure. The part marked with a rectangular frame is the location of the red fire ant nest.

[0110] S1358. In order to facilitate the remote and accurate training of the robot dog's target recognition model, a training website is built to achieve communication with the robot dog and operate the training of its model in real time.

[0111] To achieve communication with the robot dog by building a training website, the following steps can be included:

[0112] A. A lightweight website built based on the third-party python library Flask builds a server through the intranet penetration service and deploys the website to the public network;

[0113] B. Deploy the YOLOX model to the website to facilitate remote model training and visualization hyperparameter adjustment;

[0114] C. Set up the training part and the prediction part according to the division of website functions. The training part is used by technicians to evaluate the effects of different models, and the prediction part is used to detect the recognition of different models so as to select the required model for deployment according to the actual situation;

[0115] D. Based on the characteristics that the website can use IP / TCP technology to establish real-time communication with the robot dog, the images sent back by the website can be used to evaluate the field detection effect of the robot dog and provide correction feedback;

[0116] E. Based on the feedback results and combined with GPS positioning technology, a suitable exploration path is planned for the robot dog in real time, and the robot dog is remotely controlled to interact with the suspected ant nest target.

[0117] S140, setting a medication route based on the target image and its position information, and administering medication to the red fire ant nest at the corresponding position of the target image through the robot dog.

[0118] Through the combination of ecology, animal behavior and information technology, we simulated the possible locations of red fire ant nests, and conducted a visual data analysis of the ecological sites of the geographical environment of these places. The analysis results were drawn into a point cloud map and marked. Finally, this was used as the automatic navigation of the robot dog to improve its probability of identifying red fire ant nests in different ecological environments. Field surveys and red fire ant behavior studies have found that the locations of red fire ant nests have the following characteristics: 1) Red fire ant nests are located in places with direct sunlight; 2) Red fire ants cannot survive in the ecological environment of natural forests, deserts, and plateaus. 3) Places related to human activities, such as both sides of roads, lawns, and around flower beds are places where red fire ant nests are prevalent. Through existing map markings, the end point patrol inspects the possible ecological niches of red fire ants, avoids ecological niches such as forests where red fire ants cannot survive, and improves the efficiency of ant extermination.

[0119] The recognition model trained using a large number of red fire ant nest images will be placed in the robot dog. At the same time, the corresponding actions and obstacle avoidance of the recognition results will be set for the robot dog, and the sensitivity of the robot dog will be trained and improved in a special venue.

[0120] Please refer to Figure 8 As shown, S140 may specifically include the following steps:

[0121] S141, the robot dog GPS patrols the occurrence of red fire ant invasion disasters, which includes the following steps:

[0122] S1411, by arranging the path on the map, the robot dog automatically screens the nests of red fire ants according to the screening path;

[0123] S1412. Mark the GPS locations of existing red fire ant nests, and use the robot dog to continuously monitor the disinfection effect of the ant nests at different time intervals;

[0124] S1413. Automatically conduct statistical analysis on the density of red fire ant nests within a unit area and observe the migration and mating flight of ant colonies;

[0125] S142. The robot dog knocks the target ant nest with its front legs to dynamically identify whether there are forage ants and guard ants emerging from the nest, and accurately determine whether the nest is an active red fire ant nest, thereby reducing the recognition error rate to zero.

[0126] The candidate red fire ant nests found through image recognition still have an error rate of 20-30%, for example, the nests of Chinese native ants are mistaken for red fire ant nests. For another example, some red fire ant nests have all died or been abandoned, becoming empty nests. In order to more accurately determine the active invading red fire ant nests, the behavioral characteristics of red fire ants that will leave the nest in large numbers when they are vibrated are used to dynamically capture the reactions of candidate nests to disturbances. The foraging ants and guard ants of invading red fire ants are extremely sensitive to the vibrations at the entrance of the nest, and will rush out of the nest in large numbers to attack. This characteristic can be used to distinguish abandoned ant nests from native ant nests. By knocking on the target ant nest with the front legs of the robot dog, it is dynamically identified whether there are foraging ants and guard ants rushing out of the nest to accurately determine whether the nest is an active red fire ant nest, reducing the recognition error rate to zero.

[0127] S143. The robot dog uses two rounds of spraying. In the first round, borax is used to block the red fire ants in the nest and start disinfecting them. In the second round, orange oil, which is harmless to the environment, is repeatedly sprayed around the nest.

[0128] The robot dog's precision spraying reduces the amount of medicine used and the impact on the environment. Two rounds of programmed spraying are used. In the first round, borax is used to block the red fire ants in the ant nest and start disinfecting them. In the second round, orange oil (a natural product that does not pollute the environment and is highly efficient in killing red fire ants) is repeatedly sprayed around the ant nest. Through observation, after two rounds of spraying, 100% of the disinfecting effect can be achieved, and no red fire ant nest will appear again within a year.

[0129] According to the designed model, robot dogs carrying specific medicines are assigned to areas severely affected by red fire ants for precise drug delivery. During the drug delivery process, technicians make certain manual judgments based on the images returned by the robot dogs in real time, and correct any errors that may occur. The judgment and correction results are fed back to the operating robot dogs in real time so that they can adjust the detection errors based on the feedback.

[0130] The robot dog repeatedly identifies the status of the same ant nest in a certain period of multiple rounds, such as multiple times within 10 days, and completes the ant extermination behavior by spraying different types and quantities of drugs on the ant nest, ultimately achieving 100% elimination of the ants.

[0131] Embodiment 2

[0132] See also Fig. 9 , Fig. 9 Schematic diagram of a red fire ant nest identification and drug delivery device based on a robot dog disclosed in an embodiment of the present invention. Fig. 9 As shown, the red fire ant nest identification and medication device based on the robot dog may include:

[0133] The photographing unit 210 is used to take photos with the robot dog, obtain multiple images, and associate each image with corresponding position information;

[0134] A conversion unit 220, configured to convert the image into a three-dimensional scene graph;

[0135] The recognition unit 230 is used to input the three-dimensional scene graph into a pre-trained YOLOX model for recognition, and determine a target image, wherein the target image is an image corresponding to the location of a red fire ant nest;

[0136] The medication unit 240 is used to set a medication path based on the target image and its position information, and to administer medication to the red fire ant nest at the corresponding position of the target image through the robot dog.

[0137] Embodiment 3

[0138] See also Fig.10 , Fig.10 Schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Fig.10 As shown, the electronic device may include:

[0139] A memory 310 storing executable program codes;

[0140] a processor 320 coupled to the memory 310;

[0141] The processor 320 calls the executable program code stored in the memory 310 to execute part or all of the steps in the method for identifying and administering red fire ant nests based on a robot dog in the first embodiment.

[0142] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute part or all of the steps in a method for identifying and administering red fire ant nests based on a robot dog in Embodiment 1.

[0143] The embodiment of the present invention further discloses a computer program product, wherein when the computer program product is run on a computer, the computer is enabled to execute part or all of the steps in the method for identifying and administering red fire ant nests based on a robot dog in the first embodiment.

[0144] An embodiment of the present invention also discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product, wherein when the computer program product runs on a computer, the computer executes part or all of the steps in a method for identifying and administering red fire ant nests based on a robot dog in embodiment one.

[0145] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the method described in each embodiment of the present invention.

[0149] In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0150] A person of ordinary skill in the art can understand that some or all of the steps in the various methods of the embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0151] The above is a detailed introduction to a method and device for identifying and administering red fire ant nests based on a robot dog disclosed in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for identifying and administering red fire ant nests based on a robot dog, characterized in that: It includes: Taking photos with the robot dog to obtain multiple images, and associating each image with corresponding location information; converting the image into a three-dimensional scene graph; The three-dimensional scene graph is input into a pre-trained YOLOX model for recognition to determine a target image, wherein the target image is an image corresponding to the location of a red fire ant nest; Setting a medication route based on the target image and its location information, and administering medication to the red fire ant nest at the corresponding position of the target image by the robot dog; A complete three-dimensional scene graph is constructed based on the three-dimensional model of the nest and the scene where the nest is located, including: Step 11: Acquire a texture-rich environment background on site to ensure that the object and the background are relatively still during the picture shooting process; Step 12: obtain camera data by calibrating the camera, and use the above data to determine the correspondence between the feature points of the sample image and the points in the world coordinate system. The camera data includes focal length, principal point, rotation angle and translation of the camera; Step 13, matching and extracting feature points of the images, matching the feature points of sample images of the same detection target collected at different angles in pairs, screening and extracting correctly matched feature point pairs, using these feature point pairs to arrange and combine images of the same target at different angles, so that the arrangement and combination of these images show the position coordinates of various parts of the original detection target in three-dimensional space, and finally extracting the appropriate position coordinates under the images at different angles as the spatial positions of various parts of the original detection target; Step 14: According to the spatial position coordinate information of different parts of the original detection target obtained in step 13, the required foreground, i.e., the spatial parts of the original detection target, are segmented from the image set with a certain arrangement and combination obtained in step 13 by using manual annotation or image segmentation technology, and the edge contour is extracted at the same time, and all points within and on the contour line are retained, and the remaining points are eliminated to obtain the spatial set points surrounding the object; Step 15: Use the Delaunay triangulation algorithm to fit the surface of the target object using a triangular plane, project the spatial point set obtained in step 14 onto a certain plane, segment the point set on the plane, and restore the segmented triangulated network to space to achieve a triangulated model of the object, that is, a three-dimensional object model; Step 16: Use the same method to obtain a scene graph model, and combine the three-dimensional object model with the scene graph model to form a complete three-dimensional scene graph containing the object model to be identified.

2. The method for identifying and administering red fire ant nests based on a robot dog according to claim 1, characterized in that: The pre-trained YOLOX model includes: Collect sample images by web crawling and / or field photography to construct a sample image set, wherein the sample images include a plurality of images of red fire ant nests and a plurality of images of native ant nests; Using the acquired sample image set, the sample images are labeled and classified. The labeled files contain the category and location coordinate information of the ant nest. The nest target features are extracted from the sample image set according to the annotation file, and a single-camera binocular stereo vision algorithm is used to construct a three-dimensional model of the nest based on the extracted multi-angle nest features; A complete three-dimensional scene graph is constructed based on the three-dimensional model of the nest and the scene where the nest is located, and all scene graphs are aggregated to form a three-dimensional model dataset of the ant nest; The three-dimensional model dataset is constructed into a COCO input dataset of the YOLOX model standard by using model rendering and coordinate transformation to train the YOLOX model.

3. The method for identifying and administering red fire ant nests based on a robot dog according to claim 2, characterized in that: After building the sample image set, it also includes: The sample images are further processed using manual labeling and classification methods using the Labelme program. The processed images and labels are converted from the original images and label data into a standard data set of the YOLOX model through the labelme2coco program. The constructed sample image set is divided into training data and verification data of the YOLOX model through segmentation.

4. The method for identifying and administering red fire ant nests based on a robot dog according to claim 1, characterized in that: The three-dimensional model dataset is constructed into a COCO input dataset of the YOLOX model standard by using model rendering and coordinate transformation, and the YOLOX model is trained, including: Step 21, construct a model rendering function: R (Mo, Mb, po, eo, to, pw, ew, hi, wi, L); wherein R is a scene rendering function, Mo is a 3D model of the object to be measured, Mb is a scene background model, po, eo, to are respectively the center point coordinates, Euler angles, and texture parameters of Mo in M, pw, ew are respectively the center point coordinates and Euler angles of the image window in M; hi, wi are respectively the height and width of the output image, and L is the lighting condition of the three-dimensional scene graph; Step 22: The three-dimensional scene graph is passed through a rendering function multiple times to obtain an original data set, and the coordinates of the upper left vertex, the lower right vertex, and the width and height of the image of the bounding box are calculated to obtain the coordinates of the center point, width, height, and relative ratio of the image width and height of the bounding box based on the coordinates of the upper left vertex, the coordinates of the lower right vertex, and the width and height of the image of the bounding box in the marked center of the original data set; Step 23: Convert the three-dimensional scene graph into the YOLOX standard COCO dataset using the relative proportion obtained in step 22.

5. The method for identifying and administering red fire ant nests based on a robot dog according to claim 1, characterized in that: The 3D model dataset is constructed using model rendering and coordinate transformation to construct a COCO input dataset of the YOLOX model standard, and the YOLOX model is trained, including: Step 31: Input the red fire ant nest model dataset of different sizes processed into COCO format into the YOLOX model; Step 32: Pass the input data through the Backbone feature extraction network. The output of this layer of network is three effective feature layers of different sizes obtained by scaling the original image model accordingly. Compared with the original image model size H*W, the outputs of the three feature layers are H / 8*W / 8, H / 16*W / 16, and H / 32*W / 32, respectively, where H and W are the height and width of the image, respectively. Step 33: Pass the three feature layers in step 32 through the FPN layer and the PAN layer respectively to obtain outputs of three original feature map sizes, and then use the outputs as inputs of the Decouple Head layer; Step 34: In the Decouple Head layer, the feature map is separated, classified and located in the form of a decoupled head, and finally the target category, location information and IoU in the feature map are obtained through classification and regression tasks; Step 35: In order to facilitate the remote and accurate training of the robot dog's target recognition model, a training website is built to achieve communication with the robot dog and operate the training of its model in real time.

6. The method for identifying and administering red fire ant nests based on a robot dog according to claim 5, characterized in that: Communication with the robot dog is achieved by building a training website, including: Step 41: A lightweight website built based on the third-party python library Flask is used to build a server through the intranet penetration service and deploy the website to the public network; Step 42: Deploy the YOLOX model to the website to facilitate remote model training and visualization hyperparameter adjustment; Step 43: Set up a training part and a prediction part according to the division of website functions. The training part is used by technicians to evaluate the effects of different models, and the prediction part is used to detect the recognition status of different models so as to select the required model for deployment according to the actual situation. Step 44: Based on the characteristics of the website using IP / TCP technology to establish real-time communication with the robot dog, the images sent back by the website are used to evaluate the field detection effect of the robot dog and provide correction feedback; Step 45: Based on the feedback results and combined with GPS positioning technology, a suitable exploration path is planned for the robot dog in real time, and the robot dog is remotely controlled to interact with the suspected ant nest target.

7. The method for identifying and administering red fire ant nests based on a robot dog according to claim 6, characterized in that: The method of setting a medication path based on the target image and its position information, and administering medication to the red fire ant nest at the corresponding position of the target image by the robot dog, comprises: Step 51: The robot dog GPS patrols to detect the occurrence of red fire ant disasters, which includes the following steps: Step 511: By arranging the path on the map, the robot dog automatically screens the nests of red fire ants according to the screening path; Step 512: Mark the existing GPS location of the red fire ant nest, and have the robot dog continuously monitor the disinfecting effect of the ant nest at different time intervals; Step 513, automatically statistically analyzing the density of red fire ant nests within a unit area, and observing the migration and mating flight of ant colonies; Step 52: The robot dog knocks the target ant nest with its front legs to dynamically identify whether there are forage ants and guard ants emerging from the ant nest, and accurately determine whether the ant nest is an active red fire ant nest, thereby reducing the recognition error rate to zero; Step 53: The robot dog uses two rounds of spraying. In the first round, borax is used to seal the red fire ants in the nest and start disinfecting them. In the second round, orange oil, which is harmless to the environment, is repeatedly sprayed around the nest.

8. A device for identifying and administering red fire ant nests based on a robot dog, characterized in that: include: A shooting unit, used to take pictures with the robot dog, obtain multiple images, and associate each image with corresponding position information; A conversion unit, used for converting the image into a three-dimensional scene graph; A recognition unit, used for inputting the three-dimensional scene graph into a pre-trained YOLOX model for recognition, and determining a target image, wherein the target image is an image corresponding to the location of a red fire ant nest; A drug delivery unit, used to set a drug delivery path based on the target image and its position information, and deliver drugs to the red fire ant nest at the corresponding position of the target image through the robot dog; Conversion unit, including: Acquire the texture-rich environment background on site to ensure the relative stillness of the object and the background during the picture shooting process; Obtain camera data by calibrating the camera, and use the above data to determine the correspondence between the feature points of the sample image and the points in the world coordinate system, wherein the camera data includes focal length, principal point, rotation angle and translation of the camera; Match and extract feature points of images, match feature points of sample images of the same detection target collected at different angles in pairs, screen and extract correctly matched feature point pairs, use these feature point pairs to arrange and combine images of the same target at different angles, so that the arrangement and combination of these images show the position coordinates of various parts of the original detection target in three-dimensional space, and finally extract the appropriate position coordinates under images of different angles as the spatial position of various parts of the original detection target; According to the spatial position coordinate information of different parts of the original detection target obtained in step 13, the required foreground, i.e., the spatial parts of the original detection target, are segmented out from the image set with a certain arrangement and combination mode obtained in step 13 by using manual annotation or image segmentation technology, and the edge contour is extracted at the same time, and all points within and on the contour line are retained, and the remaining points are eliminated to obtain the spatial set points surrounding the object; Using the Delaunay triangulation algorithm, the surface of the target object is fitted with a triangular plane, the spatial point set obtained in step 14 is projected onto a certain plane, the point set on the plane is segmented, and the segmented triangulated network is restored to space to realize the triangulated model of the object, that is, the three-dimensional object model; The same method is used to obtain the scene graph model, and the three-dimensional object model is combined with the scene graph model to form a complete three-dimensional scene graph containing the object model to be identified.

Images