Large-scale mechanical arm livestock and poultry feeding method and device based on machine vision

Abstract

The invention relates to the technical field of livestock and poultry intelligent breeding, in particular to a large-scale mechanical arm livestock and poultry feeding method and device.The method comprises the steps that a current depth image of a livestock and poultry feed box is collected through image collecting equipment on a transfer mechanical arm, and an area of interest is intercepted from the current depth image; determining a bottom reference plane of the material box; according to the bottom reference plane and the current depth image, the amount of feed needing to be loaded into the transfer mechanical arm is calculated; smoothing a transfer mechanical arm loaded with feed to a detection pre-pose, and capturing a three-dimensional point cloud of the livestock and poultry cage through an image acquisition device so as to calculate a minimum bounding box center coordinate of the feed box; and according to the minimum bounding box center coordinate of the livestock and poultry feed box and the detected pre-pose, a space linear track is calculated, and according to the space linear track, a transfer mechanical arm loaded with feed is controlled to move to the position over the target livestock and poultry feed box, and the feed is released. Therefore, the problem that intelligent and personalized feeding is difficult to achieve by existing automatic equipment is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent livestock and poultry breeding, and particularly relates to a method and device for large-scale robotic arm livestock and poultry feeding based on machine vision. Background Art

[0002] With the rapid development of large-scale breeding, higher requirements are put forward for the accuracy and efficiency of livestock and poultry feeding. The traditional livestock and poultry feeding method mainly relies on manual operation. This method not only has extremely high labor intensity and requires a large amount of human resources, but also it is difficult to accurately control the feed intake of each livestock and poultry during the feeding process. Due to the limitations of manual operation, it is often difficult to accurately judge the actual needs of livestock and poultry, which easily causes overfeeding or underfeeding of feed, thus leading to waste of feed or nutritional imbalance of livestock and poultry. In addition, it is difficult to ensure the timeliness and uniformity of feeding by manual operation, which may affect the growth rate and health status of livestock and poultry.

[0003] Although the prior art has been improved to a certain extent, for example, using automated equipment for feeding, there are still some obvious disadvantages. For example, although some automated equipment can reduce the manual labor intensity, there is still room for improvement in the accuracy of feed feeding and the identification of individual needs of livestock and poultry. At the same time, some automated equipment has limitations in adapting to different types and different growth stages of livestock and poultry, and it is difficult to achieve true intelligent and personalized feeding. Summary of the Invention

[0004] The present invention provides a method and device for large-scale robotic arm livestock and poultry feeding based on machine vision to solve problems such as the difficulty of achieving intelligent and personalized feeding by existing automated equipment.

[0005] An embodiment of the first aspect of the present invention provides a method for large-scale robotic arm livestock and poultry feeding based on machine vision, including the following steps: collecting the current depth image of the target livestock and poultry feed box by using an image acquisition device pre-installed on the target transfer robotic arm; intercepting the region of interest in the current depth image, and extracting a plurality of data points in the region of interest to fit the plurality of data points to obtain the bottom reference plane of the target livestock and poultry feed box; calculating the remaining feed volume in the target livestock and poultry feed box according to the bottom reference plane and the current depth image; generating a supplementary feeding requirement according to the remaining feed volume and a preset feeding amount standard, and loading a corresponding amount of feed into the transfer robotic arm according to the supplementary feeding requirement; smoothly moving the transfer robotic arm loaded with feed to a detection pre-pose, and capturing the three-dimensional point cloud of the target livestock and poultry cage through the image acquisition device; collecting the point cloud cluster of the feed box in the three-dimensional point cloud of the target livestock and poultry cage, and calculating the center coordinates of the minimum bounding box of the feed box according to the point cloud cluster of the feed box; calculating the spatial straight-line trajectory according to the center coordinates of the minimum bounding box of the feed box and the detection pre-pose, and controlling the transfer robotic arm loaded with feed to move above the target livestock and poultry feed box according to the spatial straight-line trajectory, and releasing the feed into the target livestock and poultry feed box.

[0006] Optionally, the step of collecting the current depth image of the target livestock and poultry feed box by using an image acquisition device pre-installed on the target transfer robotic arm includes:

[0007] Performing checkerboard calibration on the image acquisition device pre-installed on the target transfer robotic arm to obtain the calibrated image acquisition device; collecting the current depth image of the target livestock and poultry feed box by using the calibrated image acquisition device.

[0008] Optionally, the step of intercepting the region of interest in the current depth image, and extracting a plurality of data points in the region of interest to fit the plurality of data points to obtain the bottom reference plane of the target livestock and poultry feed box includes:

[0009] Performing edge detection on the current depth image to obtain the binary edge image of the target livestock and poultry feed box; performing peak detection on the binary edge image to determine the binary boundary image of the target livestock and poultry feed box; intercepting the region of interest in the binary boundary image, and extracting a plurality of data points in the region of interest to fit the plurality of data points to obtain the bottom reference plane of the target livestock and poultry feed box.

[0010] Optionally, the step of calculating the remaining feed volume in the target livestock and poultry feed box according to the bottom reference plane and the current depth image includes:

[0011] Perform multi-frame sliding average filtering on the current depth image to obtain a smoothed depth image; calculate the remaining feed volume in the target livestock and poultry feed box according to the height of each pixel point and the bottom reference plane.

[0012] Optionally, the step of smoothly moving the transfer robotic arm loaded with feed to the detection pre-pose and capturing the three-dimensional point cloud of the target livestock and poultry cage by the image acquisition device includes:

[0013] Based on a preset joint space interpolation algorithm and RRT* algorithm, smoothly move the transfer robotic arm loaded with feed from the loading pose to the detection pre-pose; control the image acquisition device to emit encoded stripes at a preset frequency to capture the three-dimensional point cloud of the target livestock and poultry cage.

[0014] Optionally, the step of collecting the point cloud cluster of the feed box in the three-dimensional point cloud of the target livestock and poultry cage and calculating the center coordinates of the minimum bounding box of the feed box according to the point cloud cluster of the feed box includes:

[0015] Perform voxel downsampling and statistical outlier filtering on the three-dimensional point cloud of the target livestock and poultry cage to obtain the processed three-dimensional point cloud of the target livestock and poultry cage; segment the support plane in the processed three-dimensional point cloud of the target livestock and poultry cage to extract the point cloud cluster of the feed box; perform principal component analysis on the point cloud cluster of the feed box to calculate the center coordinates of the minimum bounding box of the feed box.

[0016] An embodiment of the second aspect of the present invention provides a large-scale robotic arm livestock and poultry feeding device based on machine vision, including: a collection module, configured to collect the current depth image of the target livestock and poultry feed box by using an image acquisition device pre-installed on the target transfer robotic arm; a fitting module, configured to intercept the region of interest in the current depth image and extract a plurality of data points in the region of interest to fit the plurality of data points to obtain the bottom reference plane of the target livestock and poultry feed box; a volume calculation module, configured to calculate the remaining feed volume in the target livestock and poultry feed box according to the bottom reference plane and the current depth image; a demand generation module, configured to generate a feeding supplement demand according to the remaining feed volume and a preset feeding amount standard, and load a corresponding amount of feed into the transfer robotic arm according to the feeding supplement demand; a capture module, configured to smoothly move the transfer robotic arm loaded with feed to the detection pre-pose and capture the three-dimensional point cloud of the target livestock and poultry cage by the image acquisition device; a coordinate calculation module, configured to collect the point cloud cluster of the feed box in the three-dimensional point cloud of the target livestock and poultry cage and calculate the center coordinates of the minimum bounding box of the feed box according to the point cloud cluster of the feed box; a feeding module, configured to calculate a space straight line trajectory according to the center coordinates of the minimum bounding box of the feed box and the detection pre-pose, and control the transfer robotic arm loaded with feed to move to directly above the target livestock and poultry feed box according to the space straight line trajectory and release the feed into the target livestock and poultry feed box..

[0017] In a third aspect embodiment of the present invention, an electronic device is provided, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the method for large-scale robotic arm livestock and poultry feeding based on machine vision as described in the above embodiments.

[0018] In a fourth aspect embodiment of the present invention, a computer program product is provided, and when the computer program / instructions are executed by a processor, the method for large-scale robotic arm livestock and poultry feeding based on machine vision as described above is implemented.

[0019] In a fifth aspect embodiment of the present invention, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program, and when the program is executed by a processor, the method for large-scale robotic arm livestock and poultry feeding based on machine vision as described above is implemented.

[0020] The method and device for large-scale robotic arm livestock and poultry feeding based on machine vision proposed in the embodiments of the present invention can accurately identify the remaining amount in the feed box of each livestock and poultry, so as to fully understand the feed intake of each livestock and poultry, accurately calculate and control the amount of feed fed each time, effectively avoid the problems of overfeeding or underfeeding of feed caused by inaccurate judgment in traditional manual feeding, significantly reduce feed waste, and ensure the nutritional balance of livestock and poultry at the same time; the combined use of a robotic arm and machine vision recognition not only realizes the automation of the feeding process, greatly reduces the labor intensity of manual work, but also can quickly respond to the feeding needs of livestock and poultry, ensure feeding at the best time, and improve the timeliness and efficiency of feeding; it has good adaptability and scalability, can easily meet the feeding needs of different types and different growth stages of livestock and poultry, and can continuously improve the recognition accuracy and feeding efficiency through continuous optimization and upgrade of algorithms, meeting the long-term development needs of large-scale farms.

[0021] The additional aspects and advantages of the present invention will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present invention will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0023] Figure 1 is a flowchart of a method for large-scale robotic arm livestock and poultry feeding based on machine vision provided by an embodiment of the present invention;

[0024] Figure 2 is a specific execution schematic diagram of a method for large-scale robotic arm livestock and poultry feeding based on machine vision provided by an embodiment of the present invention;

[0025] Figure 3Schematic block diagram of a large-scale robotic arm livestock feeding device based on machine vision provided by an embodiment of the present invention;

[0026] Figure 4 Schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0027] Explanation of reference numerals: 1 - transfer robotic arm, 2 - transfer feed box, 3 - image acquisition device, 4 - livestock feed box, 5 - livestock cage. Detailed implementation manners

[0028] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as a limitation of the present invention.

[0029] The large-scale robotic arm livestock feeding method and device based on machine vision according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0030] Figure 1 Schematic flow diagram of a large-scale robotic arm livestock feeding method based on machine vision provided by an embodiment of the present invention.

[0031] As Figure 1 shown, the large-scale robotic arm livestock feeding method based on machine vision includes the following steps:

[0032] In step S101, the current depth image of the target livestock feed box is acquired by using an image acquisition device pre-installed on the target transfer robotic arm.

[0033] In some embodiments, acquiring the current depth image of the target livestock feed box 4 by using the image acquisition device 3 pre-installed on the target transfer robotic arm includes:

[0034] Performing checkerboard calibration on the image acquisition device 3 pre-installed on the target transfer robotic arm to obtain the calibrated image acquisition device 3;

[0035] Acquiring the current depth image of the target livestock feed box 4 by using the calibrated image acquisition device 3.

[0036] It should be noted that, as Figure 2 shown, the target transfer robotic arm 1 is provided with a transfer feed box 2, the transfer feed box 2 is arranged at the end of the robotic arm, and the image acquisition device 3 is arranged on the transfer feed box 2.

[0037] During the actual execution process, the target transfer robotic arm 1 moves the image acquisition device 3 (such as a structured light depth camera) to a preset height (e.g., 20 cm) directly above the target livestock and poultry feed box 4 perpendicular to it, calibrates the image acquisition device 3 with a checkerboard to establish the depth mapping relationship between pixel coordinates and the three-dimensional coordinates of the camera coordinate system, obtains the calibrated image acquisition device 3, and uses the calibrated image acquisition device 3 to acquire the current depth image of the target livestock and poultry feed box 4 (i.e., the image of the remaining feed in the current target livestock and poultry feed box 4).

[0038] Furthermore, to cope with the low-light environment, an infrared fill light can be installed above the image acquisition device 3 to provide fill light for the image acquisition process, and a dust-proof cover can also be added to the image acquisition device 3 to avoid feed dust contaminating the lens.

[0039] In step S102, an area of interest is intercepted from the current depth image, and multiple data points are extracted from the area of interest to fit the multiple data points to obtain the bottom reference plane of the target livestock and poultry feed box.

[0040] In some embodiments, intercepting an area of interest from the current depth image and extracting multiple data points from the area of interest to fit the multiple data points to obtain the bottom reference plane of the target livestock and poultry feed box includes:

[0041] Performing edge detection on the current depth image to obtain the binary edge image of the target livestock and poultry feed box;

[0042] Performing peak detection on the binary edge image to determine the binary boundary image of the target livestock and poultry feed box;

[0043] Intercepting an area of interest from the binary boundary image and extracting multiple data points from the area of interest to fit the multiple data points to obtain the bottom reference plane of the target livestock and poultry feed box.

[0044] During the actual execution process, the Canny edge detection method is used to detect the edges of the current depth image to connect the weak edges connected to the strong edges, thereby obtaining the binary edge image of the target livestock and poultry feed box 4; further, the Hough transform localization method is used to detect the peaks of the binary edge image to determine the four vertex coordinates of the target livestock and poultry feed box 4 through the intersection of straight lines to complete edge localization and obtain the binary boundary image of the target livestock and poultry feed box 4; next, an interested region is intercepted in the binary boundary image, and the RANSAC algorithm is used to randomly select three data points in the interested region to fit a plane model (normal vector and intercept), and calculate the distances from all other data points to this plane model. If the distance from a data point to this plane model is less than the preset threshold, it is considered that the data point belongs to this plane model and is included in the inlier set. By iterating the above process multiple times, the plane model with the largest inlier set is selected as the bottom reference plane of the target livestock and poultry feed box 4.

[0045] In step S103, the remaining feed volume in the target livestock and poultry feed box 4 is calculated based on the bottom reference plane and the current depth image.

[0046] In some embodiments, calculating the remaining feed volume in the target livestock and poultry feed box based on the bottom reference plane and the current depth image includes:

[0047] Perform multi-frame sliding average filtering on the current depth image to obtain a smoothed depth image;

[0048] Calculate the remaining feed volume in the target livestock and poultry feed box based on the height of each pixel point and the bottom reference plane.

[0049] During the actual execution process, since the surface of the remaining feed in the target livestock and poultry feed box 4 has unevenness, the embodiments of the present invention perform multi-frame sliding average filtering on the current depth image. Specifically, each frame of the depth image is preprocessed (such as denoising, voxel downsampling, etc.), and then the depth values at the same position are averaged according to a sliding window (such as 5 frames) to obtain a smoothed depth image, thereby reducing the measurement error caused by the uneven feed surface and improving the stability and accuracy of the remaining feed volume in the target livestock and poultry feed box 4 calculated subsequently.

[0050] Further, the remaining feed volume (i.e., the remaining feed volume in the target livestock and poultry feed box 4) obtained by calculating the cumulative volume of the height differences of each pixel point based on the height of each pixel point and the bottom reference plane, the expression is: (V = ∑hi × pixel area), and in combination with the dynamic calibration mechanism, the transfer robotic arm 1 is moved below the feeding device of the total feed bin.

[0051] In step S104, a feeding requirement is generated based on the remaining feed volume and the preset feeding amount standard to load the corresponding amount of feed into the transfer robotic arm according to the feeding requirement.

[0052] As Figure 2 shown, during the actual execution process, the supplementary feeding requirement is generated based on the volume of the leftover materials and the preset feeding amount standard. By using the linear equation fitted from the test data of the outlet flow rate of the feeding equipment in the total feed bin and the supplementary feeding requirement, the feeding duration corresponding to the required supplementary amount of the feeding equipment in the total feed bin is deduced. The supplementary feeding requirement and the feeding time are sent to the feeding equipment together. The feeding equipment drives the stepper motor to control the auger rotation speed, and at the same time adopts a closed-loop feedback mechanism to feed into the transfer box 2 of the transfer robotic arm 1. A weighing device can be set inside the transfer box 2 after each feeding to weigh the feed again to verify the actual supplementary feeding amount. If the error exceeds ±2%, the coefficients of the linear equation are dynamically corrected until the corresponding amount of feed is loaded in the transfer box 2.

[0053] In step S105, the transfer robotic arm 1 loaded with feed is smoothly moved to the detection pre-pose, and the three-dimensional point cloud of the target livestock and poultry cage is captured by the image acquisition device.

[0054] In some embodiments, moving the transfer robotic arm 1 loaded with feed smoothly to the detection pre-pose and capturing the three-dimensional point cloud of the target livestock and poultry cage includes:

[0055] Based on the preset joint space interpolation algorithm and RRT* algorithm, the transfer robotic arm loaded with feed is smoothly moved from the loading position to the detection pre-pose;

[0056] Controlling the image acquisition device to emit encoded stripes at a preset frequency to capture the three-dimensional point cloud of the target livestock and poultry cage.

[0057] During the actual execution process, based on the preset joint space interpolation algorithm (such as quintic polynomial trajectory planning), the transfer robotic arm 1 loaded with feed is smoothly moved from the loading position to the detection pre-pose. During this process, the dynamic obstacles (such as the moving rabbit cage bracket) on the path are detected in real time by the RRT* algorithm to ensure that the end effector carries the box to reach the observation point at a preset moving speed at a height from the target cage position. Then, the image acquisition device 3 projects encoded stripes at a preset frequency onto the target livestock and poultry cage 5 to capture the three-dimensional point cloud of the target livestock and poultry cage.

[0058] In step S106, the box point cloud cluster is collected from the three-dimensional point cloud of the target livestock and poultry cage, and the center coordinates of the minimum bounding box of the box are calculated according to the box point cloud cluster.

[0059] In some embodiments, collecting the box point cloud cluster from the three-dimensional point cloud of the target livestock and poultry cage and calculating the center coordinates of the minimum bounding box of the box includes:

[0060] Performing voxel downsampling and statistical outlier filtering on the three-dimensional point cloud of the target livestock and poultry cage to obtain the processed three-dimensional point cloud of the target livestock and poultry cage;

[0061] The support plane in the three-dimensional point cloud of the target livestock and poultry cage after segmentation processing is segmented to extract the point cloud clusters of the feed boxes;

[0062] Principal component analysis is performed on the point cloud clusters of the feed boxes to calculate the center coordinates of the minimum bounding box of the feed boxes.

[0063] In the actual execution process, the three-dimensional point cloud of the target livestock and poultry cage is first subjected to voxel downsampling (such as a 2-mm voxel grid) and statistical outlier filtering using the PCL library to obtain the processed three-dimensional point cloud of the target livestock and poultry cage. Then, the support plane in the processed three-dimensional point cloud of the target livestock and poultry cage is segmented by the RANSAC algorithm to extract the point cloud clusters of the feed boxes. Finally, the center coordinates of the minimum bounding box of the feed boxes are calculated using principal component analysis (PCA).

[0064] In step S107, a spatial straight-line trajectory is calculated based on the center coordinates of the minimum bounding box of the feed box and the detection pre-pose, so as to control the transfer manipulator loaded with feed to move directly above the target livestock and poultry feed box according to the spatial straight-line trajectory and release the feed into the target livestock and poultry feed box.

[0065] In the actual execution process, the position of the feed box in the camera coordinate system (i.e., the detection pre-pose) is converted into the coordinates in the manipulator base coordinate system through the hand-eye calibration matrix. A Cartesian spatial straight-line trajectory of the transfer manipulator 1 is planned based on the center coordinates of the minimum bounding box of the feed box and the coordinates in the manipulator base coordinate system. The transfer manipulator 1 loaded with feed is controlled to move to a preset height directly above the target livestock and poultry feed box 4 according to the spatial straight-line trajectory. After reaching the corresponding position, the feeding gate of the transfer feed box 2 is triggered, so that the feed in the transfer feed box 2 is put into the target livestock and poultry feed box 4.

[0066] In summary, the large-scale manipulator livestock and poultry feeding method based on machine vision proposed in the embodiments of the present invention has the following beneficial effects:

[0067] (1) It can accurately identify the remaining amount in the feed box of each livestock and poultry, so as to fully understand the feed intake of each livestock and poultry, accurately calculate and control the amount of feed fed each time, effectively avoid the problems of overfeeding or underfeeding of feed caused by inaccurate judgment in traditional manual feeding, significantly reduce feed waste, and ensure the nutritional balance of livestock and poultry at the same time;

[0068] (2) The combined use of the manipulator and machine vision recognition not only realizes the automation of the feeding process, greatly reduces the manual labor intensity, but also can quickly respond to the feeding needs of livestock and poultry, ensure feeding at the best time, and improve the timeliness and efficiency of feeding;

[0069] (3) It has good adaptability and scalability, can easily meet the feeding needs of livestock and poultry of different species and different growth stages, and can continuously improve the recognition accuracy and feeding efficiency through the continuous optimization and upgrading of algorithms, so as to meet the long-term development needs of large-scale farms.

[0070] Next, a large-scale robotic arm livestock and poultry feeding device based on machine vision according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0071] Figure 3 It is a block diagram of a large-scale robotic arm livestock and poultry feeding device based on machine vision according to an embodiment of the present invention.

[0072] As Figure 3 shown, the large-scale robotic arm livestock and poultry feeding device 30 based on machine vision includes: a collection module 301, a fitting module 302, a volume calculation module 303, a demand generation module 304, a capture module 305, a coordinate calculation module 306, and a feeding module 307.

[0073] Among them, the collection module 301 is used to collect the current depth image of the target livestock and poultry feed box by using the image acquisition device pre-installed on the target transfer robotic arm. The fitting module 302 is used to intercept the region of interest in the current depth image and extract multiple data points in the region of interest to fit the multiple data points to obtain the bottom reference plane of the target livestock and poultry feed box. The volume calculation module 303 is used to calculate the remaining material volume in the target livestock and poultry feed box according to the bottom reference plane and the current depth image. The demand generation module 304 is used to generate a feeding demand according to the remaining material volume and the preset feeding amount standard, so as to load the corresponding amount of feed into the transfer robotic arm according to the feeding demand. The capture module 305 is used to smoothly move the transfer robotic arm loaded with feed to the detection pre-pose and capture the three-dimensional point cloud of the target livestock and poultry cage through the image acquisition device. The coordinate calculation module 306 is used to collect the point cloud cluster of the feed box in the three-dimensional point cloud of the target livestock and poultry cage and calculate the center coordinate of the minimum bounding box of the feed box according to the point cloud cluster of the feed box. The feeding module 307 is used to calculate the space straight line trajectory according to the center coordinate of the minimum bounding box of the feed box and the detection pre-pose, so as to control the transfer robotic arm loaded with feed to move to directly above the target livestock and poultry feed box according to the space straight line trajectory and release the feed into the target livestock and poultry feed box.

[0074] In some embodiments, the collection module 301 includes:

[0075] A calibration unit for performing checkerboard calibration on the image acquisition device pre-installed on the target transfer robotic arm to obtain the calibrated image acquisition device;

[0076] A collection unit for collecting the current depth image of the target livestock and poultry feed box by using the calibrated image acquisition device.

[0077] In some embodiments, the fitting module 302 includes:

[0078] A first detection unit for performing edge detection on the current depth image to obtain a binary edge image of the target livestock and poultry feed box;

[0079] A second detection unit for performing peak detection on the binary edge image to determine a binary boundary image of the target livestock and poultry feed box;

[0080] A fitting unit for intercepting a region of interest in the binary boundary image and extracting multiple data points in the region of interest to fit the multiple data points to obtain the bottom reference plane of the target livestock and poultry feed box.

[0081] In some embodiments, the volume calculation module 303 includes:

[0082] An image processing unit for performing multi-frame sliding average filtering on the current depth image to obtain a smoothed depth image;

[0083] A volume calculation unit for calculating the remaining feed volume in the target livestock and poultry feed box according to the height of each pixel point and the bottom reference plane.

[0084] In some embodiments, the capture module 305 includes:

[0085] A motion unit for smoothly moving the transfer robotic arm loaded with feed from the loading position to the detection pre-pose based on a preset joint space interpolation algorithm and RRT* algorithm;

[0086] A capture unit for controlling the image acquisition device to emit encoded stripes at a preset frequency to capture the three-dimensional point cloud of the target livestock and poultry cage.

[0087] In some embodiments, the coordinate calculation module 306 includes:

[0088] A point cloud processing unit for performing voxel downsampling and statistical outlier filtering on the three-dimensional point cloud of the target livestock and poultry cage to obtain the processed three-dimensional point cloud of the target livestock and poultry cage;

[0089] A segmentation unit for segmenting the support plane in the processed three-dimensional point cloud of the target livestock and poultry cage to extract the feed box point cloud cluster;

[0090] An analysis and processing unit for performing principal component analysis on the feed box point cloud cluster to calculate the center coordinates of the minimum bounding box of the feed box.

[0091] It should be noted that the foregoing explanation of the embodiments of the large-scale robotic arm livestock and poultry feeding method based on machine vision also applies to the large-scale robotic arm livestock and poultry feeding device of this embodiment, and will not be elaborated here.

[0092] The large-scale robotic arm livestock and poultry feeding device based on machine vision proposed according to the embodiments of the present invention has the following beneficial effects:

[0093] (1) It can accurately identify the remaining amount in the feed box of each livestock and poultry, so as to fully understand the feeding of each livestock and poultry, accurately calculate and control the amount of feed fed each time, effectively avoiding the problems of overfeeding or underfeeding of feed caused by inaccurate judgment in traditional manual feeding, significantly reducing feed waste, and ensuring the nutritional balance of livestock and poultry at the same time;

[0094] (2) The combined use of the robotic arm and machine vision recognition not only realizes the automation of the feeding process, greatly reducing the labor intensity of manual work, but also can quickly respond to the feeding needs of livestock and poultry, ensuring feeding at the best time and improving the timeliness and efficiency of feeding;

[0095] (3) It has good adaptability and scalability, can easily meet the feeding needs of different types and different growth stages of livestock and poultry, and can continuously improve the recognition accuracy and feeding efficiency through continuous optimization and upgrading of algorithms, meeting the long-term development needs of large-scale farms.

[0096] Figure 4 The following is a schematic structural diagram of the electronic device provided by the embodiments of the present invention. The electronic device may include:

[0097] A memory 401, a processor 402, and a computer program stored on the memory 401 and executable on the processor 402.

[0098] When the processor 402 executes the program, it implements the large-scale robotic arm livestock and poultry feeding method based on machine vision provided in the above embodiments.

[0099] Furthermore, the electronic device further includes:

[0100] A communication interface 403 for communication between the memory 401 and the processor 402.

[0101] The memory 401 is used to store a computer program executable on the processor 402.

[0102] The memory 401 may include a high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0103] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a thick line is used in Figure 4 , but it does not mean that there is only one bus or one type of bus.

[0104] Optionally, in a specific implementation, if the memory 401, the processor 402, and the communication interface 403 are integrated on a single chip, the memory 401, the processor 402, and the communication interface 403 can communicate with each other through an internal interface.

[0105] The processor 402 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0106] The embodiments of the present invention also provide a computer program product. When the computer program / instructions are executed by a processor, the above-described method for large-scale robotic arm livestock feeding based on machine vision is implemented.

[0107] The embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above-described method for large-scale robotic arm livestock feeding based on machine vision is implemented.

[0108] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0109] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0110] Any process or method description shown in a flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or N executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0111] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or N wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0112] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0113] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0114] In addition, in each embodiment of the present invention, each functional unit can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0115] The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disc, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A large-scale robotic arm livestock and poultry feeding method based on machine vision, characterized in that: The following steps are involved: The current depth image of the target livestock and poultry feed box is collected by using an image acquisition device pre-installed on the target transport robot arm; Intercepting a region of interest in the current depth image, and extracting a plurality of data points in the region of interest, so as to fit the plurality of data points to obtain a bottom reference plane of the target livestock and poultry feed box; Calculate the remaining material volume in the target livestock and poultry feed box according to the bottom reference plane and the current depth image; Generate a feeding requirement according to the remaining material volume and a preset feeding amount standard, so as to load a corresponding amount of feed into the transfer mechanical arm according to the feeding requirement; The transfer robot arm loaded with feed is smoothly moved to the detection pre-position, and the three-dimensional point cloud of the target livestock and poultry cage is captured by the image acquisition device; Collecting a material box point cloud cluster in the target livestock and poultry cage three-dimensional point cloud, and calculating the center coordinates of the minimum bounding box of the material box according to the material box point cloud cluster; A spatial linear trajectory is calculated based on the center coordinates of the minimum bounding box of the feed box and the detected pre-pose, so as to control the transfer robot arm loaded with feed to move to just above the target livestock and poultry feed box according to the spatial linear trajectory and release the feed into the target livestock and poultry feed box.

2. The large-scale robotic arm livestock and poultry feeding method based on machine vision according to claim 1 is characterized in that: The method of collecting the current depth image of the target livestock and poultry feed box by using an image acquisition device pre-installed on the target transport mechanical arm includes: Performing chessboard calibration on the image acquisition device pre-installed on the target transport robot arm to obtain a calibrated image acquisition device; The calibrated image acquisition device is used to acquire a current depth image of the target livestock and poultry feed box.

3. The large-scale robotic arm livestock and poultry feeding method based on machine vision according to claim 1 is characterized in that: The method of intercepting a region of interest in the current depth image and extracting a plurality of data points in the region of interest to fit the plurality of data points to obtain a bottom reference plane of the target livestock and poultry feed box includes: Performing edge detection on the current depth image to obtain a binary edge image of the target livestock and poultry feed box; Performing peak detection on the binary edge image to determine the binary boundary image of the target livestock and poultry feed box; The region of interest is intercepted in the binary boundary image, and a plurality of data points are extracted from the region of interest, so as to fit the plurality of data points to obtain a bottom reference plane of the target livestock and poultry feed box.

4. The large-scale robotic arm livestock and poultry feeding method based on machine vision according to claim 1 is characterized in that: The step of calculating the residual material volume in the target livestock and poultry feed box according to the bottom reference plane and the current depth image comprises: Performing a multi-frame sliding average filtering process on the current depth image to obtain a smoothed depth image; The volume of residual material in the target livestock and poultry feed box is calculated according to the height of each pixel point and the bottom reference plane.

5. The large-scale robotic arm livestock and poultry feeding method based on machine vision according to claim 1 is characterized in that: The method of smoothly moving the transport robot arm loaded with feed to the detection pre-position and capturing the three-dimensional point cloud of the target livestock and poultry cage by the image acquisition device comprises: Based on the preset joint space interpolation algorithm and RRT* algorithm, the transport robot arm loaded with feed is smoothly moved from the loading position to the detection pre-position; The image acquisition device is controlled to emit coded stripes of a preset frequency to capture the three-dimensional point cloud of the target livestock and poultry cage.

6. The large-scale robotic arm livestock and poultry feeding method based on machine vision according to claim 1 is characterized in that: The collecting of the material box point cloud cluster in the target livestock cage three-dimensional point cloud, and calculating the center coordinates of the minimum bounding box of the material box according to the material box point cloud cluster, includes: Performing voxel downsampling and statistical outlier filtering processing on the target livestock and poultry cage three-dimensional point cloud to obtain a processed target livestock and poultry cage three-dimensional point cloud; Segmenting the support plane in the processed target livestock cage three-dimensional point cloud to extract the feed box point cloud cluster; The principal component analysis is performed on the material box point cloud cluster to calculate the center coordinates of the minimum bounding box of the material box.

7. A large-scale robotic arm livestock and poultry feeding device based on machine vision, characterized in that: include: An acquisition module, used to acquire a current depth image of a target livestock and poultry feed box using an image acquisition device pre-installed on a target transport robot arm; A fitting module, used to intercept an area of ​​interest in the current depth image, and extract a plurality of data points in the area of ​​interest, so as to fit the plurality of data points to obtain a bottom reference plane of the target livestock and poultry feed box; A volume calculation module, used for calculating the volume of residual material in the target livestock and poultry feed box according to the bottom reference plane and the current depth image; A demand generation module, used for generating a feeding demand according to the remaining feed volume and a preset feeding amount standard, so as to load a corresponding amount of feed into the transfer mechanical arm according to the feeding demand; A capture module, used to smoothly move the transfer robot arm loaded with feed to the detection pre-position, and capture the three-dimensional point cloud of the target livestock and poultry cage through the image acquisition device; A coordinate calculation module, used to collect a material box point cloud cluster in the target livestock and poultry cage three-dimensional point cloud, and calculate the coordinates of the center of the material box minimum bounding box according to the material box point cloud cluster; The feeding module is used to calculate the spatial linear trajectory according to the center coordinates of the minimum bounding box of the feed box and the detected pre-pose, so as to control the transfer robot arm loaded with feed to move to the top of the target livestock and poultry feed box according to the spatial linear trajectory, and release the feed into the target livestock and poultry feed box.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large-scale robotic arm livestock and poultry feeding method based on machine vision as described in any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program / instruction is executed by a processor, the large-scale robotic arm livestock and poultry feeding method based on machine vision as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the large-scale robotic arm livestock and poultry feeding method based on machine vision as described in any one of claims 1 to 6.

Images