A method and system for collecting cattle and sheep field leftovers based on image recognition

Abstract

The application provides a kind of based on image recognition's cattle and sheep field surplus material collection method and system, comprising: the image segmentation processing of the surplus material image obtained, using the segmentation algorithm based on region growth, according to the starting point and growth criterion of region growth self-adaptive selection according to surplus material color texture feature, obtain the surplus material image region after segmentation;For each surplus material image set, adopt edge detection algorithm to extract surplus material contour, obtain surplus material shape polygon feature by contour approximation, calculate the length, width and area size information such as surplus material according to polygon feature;Surplus material type, stain area, size parameter and entry size parameter are input into neural network model, and the motion trajectory control instruction of collection device is generated by model prediction, to control collection device to execute surplus material collection operation.The application can accurately process complex surplus material scene, realize efficient surplus material recycling.

Description

Technical Field

[0001] This invention belongs to the field of waste material recycling technology, and in particular relates to a method and system for collecting waste materials in cattle and sheep farms based on image recognition. Background Technology

[0002] In the process of collecting leftover feed in cattle and sheep farms, the shape, size, and material of the leftover feed vary greatly, making it difficult for collection devices to adapt to the collection needs of different types of leftover feed. If the parameters of the collection device are not set properly, it will affect the collection efficiency and quality. For example, if the inlet size of the collection device is set too small for large leftover feed, the leftover feed will not be able to enter smoothly, resulting in incomplete collection; while for loose and small leftover feed, if the suction force of the collection device is too large, it may suck the leftover feed into the device, causing blockage.

[0003] Meanwhile, due to the complex environment and variable lighting conditions in cattle and sheep farms, the surface of leftover materials may be covered with stains, making it difficult for image recognition systems to accurately identify the characteristic parameters of the leftover materials. Especially when leftover materials are piled up or obscured by other debris, the information acquired by the image recognition system is incomplete, affecting recognition accuracy. Furthermore, the varying distribution and orientation of leftover materials place higher demands on the orientation estimation of the image recognition system.

[0004] Therefore, how to accurately obtain the shape, size, and material characteristics of the waste material based on the image recognition results, and dynamically adjust the structure and operating parameters of the collection device to achieve adaptive collection of various types of waste material, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method and system for collecting leftover feed in cattle and sheep farms based on image recognition, thereby resolving the issues present in the prior art.

[0006] To achieve the above objectives, the present invention provides a method for collecting leftover feed in cattle and sheep farms based on image recognition, comprising the following steps:

[0007] Multi-view, multi-spectral images of leftover materials in a cattle and sheep farm environment are acquired. The leftover materials in the multi-view, multi-spectral images are then three-dimensionally completed using 3D reconstruction technology to obtain a completed image.

[0008] The region growing-based segmentation algorithm is used to segment the completed image to obtain the segmented residual material image region.

[0009] Extract feature parameters from the image region of scrap material, classify the scrap material according to the feature parameters, and obtain image sets of different types of scrap material; the feature parameters include color histogram, Gabor texture features, and Hu invariant moments;

[0010] Extract the outline of the remaining material from the image set to obtain the size information of the remaining material;

[0011] The surface stains of the scrap material were analyzed, and the stained areas were marked using the region growing method to obtain the marked image of the scrap material stains.

[0012] Construct a feature database of scrap materials, calculate the similarity between scrap material samples, update the cluster centers through the similarity matrix, and obtain the clustered feature database of scrap materials.

[0013] The feature parameters of the scrap material to be identified are input into the support vector machine model to obtain the predicted label. The predicted label is then matched with the category label in the clustered scrap material feature database to obtain the scrap material type.

[0014] The size of the collection device inlet is dynamically adjusted based on the size information of the remaining material.

[0015] The type of waste material, the area of ​​stain, the size information and the size of the inlet are input into a pre-trained neural network model. The model predicts and generates motion trajectory control commands for the collection device, which then controls the collection device to perform waste material collection operations.

[0016] Preferably, the three-dimensional completion of the remaining material in the multi-view multispectral image includes:

[0017] Multi-view, multispectral images of leftover materials in a cattle and sheep farm environment are acquired. These images are preprocessed to remove noise. A feature extraction algorithm is used to extract texture and color features of the leftover materials from the noise-removed images. Based on the extracted features, a 3D reconstruction algorithm is used to obtain the 3D spatial structure information of the leftover materials. The 3D size features of the leftover materials are extracted from this 3D spatial structure information. By analyzing the 3D spatial relationship of the leftover materials, it is determined whether there is any stacking or occlusion. If so, a 3D completion algorithm is used to reconstruct the 3D areas of the identified occluded materials to obtain complete 3D structural information of the leftover materials.

[0018] Preferably, segmenting the completed image includes:

[0019] By analyzing the color distribution and texture changes in the waste material area, the starting point and growth conditions of the region growth algorithm are determined. Starting from the selected starting point, region growth is performed according to the growth conditions. Pixels with similar colors and textures are gradually merged into the same region until no further growth is possible. The grown region is then post-processed, including region merging and hole filling, to obtain the complete waste material area.

[0020] Preferably, obtaining an image set of different types of scrap includes:

[0021] The feature parameters of the extracted scrap material image region include color histogram, Gabor texture features, and Hu invariant moments. A similarity matrix between scrap material samples is constructed based on the similarity measure of the feature parameters. The scrap material is classified using a spectral clustering algorithm. The optimal number of cluster categories is determined by the maximum modularity criterion, and image sets of different types of scrap material are obtained.

[0022] Preferably, obtaining the remaining material size information includes:

[0023] The Canny edge detection algorithm is used to extract edges from the image to obtain the edge contour image of the scrap material; the polygon features of the scrap material contour are obtained through a contour approximation algorithm; the geometric properties of the polygon are calculated based on the polygon features, including the number of sides, side length, and interior angles of the polygon; the size information of the scrap material is calculated based on the geometric properties of the polygon.

[0024] Preferably, analyzing the surface stains of the remaining material includes:

[0025] Convert the image of the scrap material to a grayscale image; calculate the grayscale histogram of the grayscale image and determine the peak and valley values ​​in the histogram; determine whether there are multiple peaks in the histogram, and if so, obtain the valley value between adjacent peaks; compare the valley value with a preset threshold, and if the valley value is lower than the threshold, determine that there are stains on the surface of the scrap material.

[0026] The present invention also provides an image recognition-based system for collecting leftover feed in cattle and sheep farms, comprising:

[0027] The image completion module is used to acquire multi-view, multi-spectral images of leftover materials in the cattle and sheep farm environment. It uses 3D reconstruction technology to complete the leftover materials in the multi-view, multi-spectral images in 3D to obtain a completed image.

[0028] The image segmentation module is used to segment the completed image based on a region growing segmentation algorithm to obtain the segmented residual image region;

[0029] The scrap material classification module is used to extract feature parameters of scrap material image regions, classify scrap material according to feature parameters, and obtain image sets of different types of scrap material; the feature parameters include color histogram, Gabor texture features, and Hu invariant moments.

[0030] The size measurement module is used to extract the outline of the leftover material from the image set to obtain the size information of the leftover material.

[0031] The stain detection module is used to analyze the stains on the surface of the waste material and to mark the stained areas using the region growing method to obtain the marked image of the waste material stains.

[0032] The clustering update module is used to construct a scrap material feature database, calculate the similarity between scrap material samples, update the cluster centers through the similarity matrix, and obtain the clustered scrap material feature database.

[0033] The matching module is used to input the feature parameters of the scrap to be identified into the support vector machine model to obtain the predicted label, and then match the predicted label with the category label in the clustered scrap feature database to obtain the scrap type.

[0034] The inlet size adjustment module is used to dynamically adjust the inlet size of the collection device based on the residual material size information;

[0035] The control module is used to input the type of waste material, the stained area, the size information and the inlet size into the pre-trained neural network model. The model predicts and generates motion trajectory control commands for the collection device, which then controls the collection device to perform waste material collection operations.

[0036] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0037] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0038] The present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0039] Compared with the prior art, the present invention has the following advantages and technical effects:

[0040] This invention provides a method and system for collecting waste materials in cattle and sheep farms based on image recognition. For complex scenarios where waste materials are piled up and obscured, multi-view multispectral imaging and 3D reconstruction techniques are used to obtain the spatial structure information of the waste materials, achieving 3D completion of obscured waste materials. Through image segmentation, feature extraction, and cluster analysis, a waste material feature database is established to identify and classify different types of waste materials. Combining stain detection and size measurement, the inlet size of the collection device is dynamically adjusted, and an optimal collection trajectory is generated. This invention integrates computer vision, machine learning, and other technologies, solving the problems of difficult identification, classification, and collection in cattle and sheep farm waste material collection, improving the intelligence and automation level of waste material collection, and has significant practical value. This invention can accurately handle complex waste material scenarios and achieve efficient waste material recycling. Attached Figure Description

[0041] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0042] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0045] Example 1

[0046] like Figure 1 As shown, this embodiment provides a method for collecting leftover feed in cattle and sheep farms based on image recognition, including the following steps:

[0047] Multi-view, multi-spectral images of leftover materials in a cattle and sheep farm environment are acquired. The leftover materials in the multi-view, multi-spectral images are then three-dimensionally completed using 3D reconstruction technology to obtain a completed image.

[0048] A region-growing-based segmentation algorithm is used to segment the completed image to obtain the segmented residual image region.

[0049] Extract feature parameters from the image region of scrap material, classify the scrap material according to the feature parameters, and obtain image sets of different types of scrap material; the feature parameters include color histogram, Gabor texture features, and Hu invariant moments;

[0050] Extract the outline of the remaining material from the image set to obtain the size information of the remaining material;

[0051] The surface stains of the scrap material were analyzed, and the stained areas were marked using the region growing method to obtain the marked image of the scrap material stains.

[0052] Construct a feature database of scrap materials, calculate the similarity between scrap material samples, update the cluster centers through the similarity matrix, and obtain the clustered feature database of scrap materials.

[0053] The feature parameters of the scrap material to be identified are input into the support vector machine model to obtain the predicted label. The predicted label is then matched with the category label in the clustered scrap material feature database to obtain the scrap material type.

[0054] The size of the collection device inlet is dynamically adjusted based on the size information of the remaining material.

[0055] The type of waste material, the area of ​​stain, the size information and the size of the inlet are input into a pre-trained neural network model. The model predicts and generates motion trajectory control commands for the collection device, which then controls the collection device to perform waste material collection operations.

[0056] Specifically, the following steps are included:

[0057] S101. Obtain multi-view, multispectral images of leftover materials in the cattle and sheep farm environment. For the situation of leftover material stacking and occlusion, use three-dimensional reconstruction technology to obtain the spatial structure information of leftover materials, extract the three-dimensional size features of leftover materials, determine the stacking situation of leftover materials by analyzing the three-dimensional spatial position relationship of leftover materials, and perform three-dimensional completion of the occluded leftover materials.

[0058] Specifically, multi-view, multispectral images of leftover materials in the cattle and sheep farm environment are acquired; these images are preprocessed to remove noise and improve image quality; feature extraction algorithms are used to extract texture, color, and other features of the leftover materials from the multi-view, multispectral images; based on the extracted features, a 3D reconstruction algorithm is used to obtain the 3D spatial structure information of the leftover materials; 3D size features of the leftover materials are extracted from the 3D spatial structure information; by analyzing the 3D spatial positional relationship of the leftover materials, it is determined whether there is any stacking or occlusion; if so, the occluded areas are identified; for the identified occluded areas, a 3D completion algorithm is used for 3D reconstruction to obtain complete 3D structural information of the leftover materials.

[0059] S102. Perform image segmentation processing on the acquired scrap material image. Use a region growing-based segmentation algorithm to adaptively select the starting point and growth criteria of region growing according to the color and texture features of the scrap material to obtain the segmented scrap material image region.

[0060] Specifically, based on the color and texture features of the leftover material image, the starting point and growth criteria of the region growing algorithm are adaptively selected. By analyzing the color distribution and texture changes in the leftover material region, the optimal starting point position and growth conditions are determined. Starting from the selected starting point, region growing is performed according to the growth criteria. Pixels with similar colors and textures are gradually merged into the same region until no further growth is possible. During the region growing process, an adaptive threshold control strategy is adopted, dynamically adjusting the growth threshold according to the color and texture distribution of pixels within the region to improve segmentation accuracy and robustness. Post-processing is performed on the grown region, including region merging and hole filling, to eliminate over-segmentation and under-segmentation, resulting in a complete leftover material region.

[0061] S103. Extract the color histogram, Gabor texture features, and Hu invariant moments of the scrap material image region. Construct a similarity matrix between scrap material samples based on the similarity measure of the feature parameters. Use spectral clustering algorithm to classify the scrap materials. Determine the optimal number of cluster categories through the maximum modularity criterion to obtain image sets of different types of scrap materials.

[0062] Specifically, for scrap material images, color histogram features, Gabor texture features, and Hu invariant moment features are extracted to construct feature vectors for the scrap material images. Based on the feature vectors, the similarity between scrap material samples is calculated using Euclidean distance, constructing a similarity matrix for the scrap material samples. This similarity matrix is ​​used as input to a spectral clustering algorithm. By solving for the eigenvalues ​​and eigenvectors of the Laplacian matrix, the scrap material samples are mapped to a low-dimensional space. In this low-dimensional space, the K-means clustering algorithm is used to cluster the scrap material samples. The optimal number of clusters is determined using the maximum modularity criterion, yielding the clustering results. Based on the clustering results, image sets of different types of scrap material are obtained, and the images of each category are visualized and analyzed. For different types of scrap material image sets, convolutional neural networks are used for feature extraction and representation learning to construct deep features for the scrap material images, which are then used for subsequent scrap material recognition and matching tasks.

[0063] S104. For each type of scrap image set, the Canny edge detection algorithm is used to extract the scrap outline. The polygonal features of the scrap shape are obtained by approximating the outline. The length, width and area of ​​the scrap are calculated based on the polygonal features.

[0064] Specifically, the input scrap material image is preprocessed using a Gaussian filtering algorithm to smooth and reduce noise interference. Based on the preprocessed scrap material image, the Canny edge detection algorithm is used to extract edges, obtaining the edge contour image of the scrap material. Contour analysis is performed on the extracted scrap material edge contour image, and a contour approximation algorithm is used to obtain the polygonal features of the scrap material contour, simplifying the shape representation of the contour. Based on the polygonal features of the scrap material contour, the geometric properties of the polygon are calculated, including the number of sides, side lengths, interior angles, and other shape feature parameters. Based on the geometric properties of the polygon, the length, width, and other dimensional information of the scrap material are further calculated using the minimum bounding rectangle or minimum bounding circle method. Statistical analysis is performed on the calculated scrap material dimensions, calculating the average length, width, and area, which serve as the basis for scrap material classification. A Support Vector Machine (SVM) algorithm is used to classify the scrap material; the SVM classifier is trained based on the dimensional statistical features of the scrap material to achieve automatic classification and recognition of the scrap material.

[0065] S105. Analyze the peak and valley features of the grayscale histogram of the scrap material image. If there are multiple peaks in the histogram and the valley value between the peaks is lower than the preset threshold, it is determined that there are stains on the surface of the scrap material. The stain area is marked by the region growing method to obtain the marked scrap material stain image.

[0066] Specifically, the process involves acquiring an image of the scrap material to be inspected and converting it into a grayscale image; calculating the grayscale histogram of the grayscale image to determine the peaks and valleys in the histogram; determining whether multiple peaks exist in the histogram, and if so, obtaining the valleys between adjacent peaks; comparing the valleys with a preset threshold, and if the valleys are below the threshold, determining that there are stains on the surface of the scrap material; marking the stained areas using a region growing method, and superimposing the marking results onto the original scrap material image to obtain a marked scrap material stain image; determining the location, size, shape, and other feature parameters of the stained areas based on the marked scrap material stain image; and outputting the feature parameters of the stained areas as the basis for subsequent process handling, so as to perform operations such as cleaning or removal of the scrap material.

[0067] S106. Construct a feature database of scrap materials. Select scrap material samples as cluster centers, calculate the similarity between scrap material samples by comprehensively considering features such as color, texture, shape and size of the scrap materials, update the cluster centers through the similarity matrix, and iterate continuously until the cluster centers no longer change, thereby obtaining the clustered feature database of scrap materials.

[0068] Specifically, the algorithm acquires features such as color, texture, shape, and size of scrap material samples to construct scrap material feature vectors; randomly selects initial cluster centers and calculates the similarity between each scrap material sample and the cluster center; classifies the scrap material samples into the category of the nearest cluster center based on the similarity; recalculates the cluster centers for the scrap material samples in each category to obtain new cluster centers; determines whether the new cluster centers have changed from the previous cluster centers; if the change is less than a preset threshold, the clustering converges and the algorithm ends; otherwise, it returns to step 3 to continue iterating; analyzes the clustering results, extracts typical scrap material samples of each category as representatives of that category, and constructs a clustered scrap material feature database; matches the new scrap material samples with the typical samples in the scrap material feature database based on similarity to determine their category, thus achieving rapid classification and management of scrap materials.

[0069] S107. Input the feature parameters of the scrap to be identified into the support vector machine model, obtain the category label of the scrap through model prediction, match the predicted label with the category label in the clustered scrap feature database, and find the most similar scrap type.

[0070] Specifically, the process involves obtaining the feature parameters of the scrap material to be identified and inputting these parameters into a pre-built support vector machine (SVM) model. The SVM model then performs classification prediction on the input feature parameters to obtain the predicted category label for the scrap material. Based on the predicted category label, the corresponding category is retrieved from a pre-clustered scrap material feature database. If a category matching the predicted category label is found in the scrap material feature database, the scrap material feature data under that category is extracted. The similarity between the feature parameters of the scrap material to be identified and the extracted scrap material feature data is calculated, and the scrap material type with the highest similarity is selected. The scrap material type with the highest similarity is determined as the final identification result of the scrap material to be identified, and this scrap material type is output. If no category matching the predicted category label is found in the scrap material feature database, the identification is considered a failure, and the user is prompted to re-obtain the feature parameters of the scrap material to be identified.

[0071] S108. Dynamically adjust the inlet size of the collection device based on the residual material size information, establish a neural network mapping model between the residual material size parameters and the inlet size, take the length, width and area of ​​the residual material as input to the neurons, calculate the inlet size parameters through the weights of the hidden layer neurons, and optimize the parameters on the training samples.

[0072] Specifically, the process involves acquiring the dimensions of the scrap material, including its length, width, and area; determining the input layer neurons of the neural network model based on this dimension, corresponding to the scrap material's length, width, and area; using the pre-trained neural network model and the scrap material dimension data from the input layer neurons, calculating the weights of the hidden layer neurons to obtain the inlet size parameter of the collection device corresponding to the output layer; determining whether the calculated inlet size parameter meets the preset optimization conditions; if it does, the parameter is taken as the optimal inlet size; otherwise, returning to step three for iterative optimization; controlling the inlet adjustment mechanism of the collection device based on the determined optimal inlet size parameter to dynamically adjust the inlet size to match the scrap material dimension; installing a photoelectric sensor at the inlet of the collection device to detect the time and location of the scrap material entering and transmitting the detection signal to the control unit; and the control unit determining whether the scrap material has successfully entered the collection device based on the received scrap material entry signal. If any abnormalities such as jamming or falling occur, an alarm signal is issued to notify the staff for handling.

[0073] S109. Input the type of waste material, the stained area, the size parameters and the inlet size parameters into the neural network model, and generate motion trajectory control instructions for the collection device through model prediction to control the collection device to perform waste material collection operation.

[0074] Specifically, information such as the type of waste material, the area of ​​soiling, size parameters, and inlet size parameters are obtained as input data for the neural network model. This input data is fed into the pre-trained neural network model, and the predicted trajectory of the collection device is obtained through forward propagation. Based on the predicted trajectory, the specific motion control command sequence for the collection device is determined. This sequence is transmitted to the control unit of the collection device, controlling it to perform waste material collection according to the specified trajectory. During the collection process, sensors collect real-time motion status data and waste material collection information. The collected real-time data is compared with the predicted trajectory, and the deviation is calculated. If the deviation exceeds a preset threshold, the parameters of the neural network model are fine-tuned. The actual motion data and collection results collected during the process are used to incrementally train the neural network model, continuously improving its prediction accuracy and adaptability.

[0075] In this embodiment, multi-view, multispectral images of leftover materials in the cattle and sheep farm environment are acquired. This step is to comprehensively capture information about the leftover materials. Multi-view refers to shooting from different angles, which avoids omissions caused by obstruction or a single perspective. Multispectral imaging provides information beyond the visible light range, such as infrared and ultraviolet bands, which helps in analyzing the material and composition of the leftover materials. For example, four cameras can be set up, located at the four corners of the cattle and sheep farm, each equipped with multiple filters, to acquire images in the visible light, near-infrared, and ultraviolet bands respectively. This allows the reflectance characteristics of the leftover materials under different spectra, helping to distinguish different materials such as feed residue and plastic film. The acquired multi-view, multispectral images are preprocessed to remove image noise and improve image quality. Images may be affected by factors such as changes in lighting and sensor noise during acquisition, leading to a decrease in quality. Preprocessing includes steps such as filtering and noise reduction, and lighting correction, aiming to improve image clarity and contrast. For example, Gaussian filtering is used to remove random noise in the image, and histogram equalization is used to enhance image contrast, making the edges and textures of the leftover materials clearer. These preprocessing techniques effectively improve image quality, laying a solid foundation for subsequent feature extraction and 3D reconstruction. Feature extraction algorithms are used to extract texture, color, and other features of the scrap material from multi-view, multispectral images. These features are crucial for identifying and analyzing the scrap material. Texture features describe the surface structure of the image, such as roughness and directionality; color features reflect the reflectivity of the scrap material. Gray-level co-occurrence matrices can be used to extract texture features of the scrap material, such as energy and contrast, which reflect the smoothness and uniformity of the scrap material surface. Color features can be represented using color histograms or color moments, for example, by calculating the mean and variance of the scrap material's color across various spectral bands to distinguish different types and states of scrap material. Based on the extracted scrap material features, 3D reconstruction algorithms are used to obtain the 3D spatial structure information of the scrap material. The purpose of this step is to recover the 3D shape of the scrap material from the 2D image. Common 3D reconstruction algorithms include stereo vision, structured light, and motion-based reconstruction. For example, using stereo vision technology, corresponding points in images from different viewpoints are matched, and the 3D coordinates of these points are calculated to generate a 3D point cloud of the scrap material. Point cloud data can intuitively display the spatial morphology of leftover materials, providing data support for subsequent analysis. Three-dimensional dimensional features of the leftover materials are extracted from their three-dimensional spatial structure information. These features are important parameters describing the geometry of the leftover materials, including length, width, height, and volume. These features can be directly calculated from the three-dimensional model or point cloud data of the leftover materials. For example, the length, width, and height of the leftover materials can be obtained by calculating the minimum bounding box of the point cloud data. By voxelizing the point cloud and counting the number of voxels, the volume of the leftover materials can be calculated. These dimensional features can be used to assess the quantity and distribution of leftover materials, assisting farm management. By analyzing the three-dimensional spatial relationship of the leftover materials, it can be determined whether there is any stacking or occlusion; if so, the occluded areas of the leftover materials are identified.Because scrap materials often accumulate together, occlusion occurs, affecting the acquisition of complete information about them. By analyzing the relative positional relationships between the scrap material point clouds, occlusion areas can be identified. For example, if a portion of one scrap material point cloud is occluded by another, the density of that portion will be significantly lower than the others, and its edge contour may be incomplete. These features can be used to detect occluded areas. For the identified occluded scrap material areas, a 3D completion algorithm is used for 3D reconstruction to obtain complete 3D structural information. The purpose of completion is to restore the 3D structure of the occluded area, making the 3D model of the scrap material more complete. Commonly used 3D completion methods include geometric rule-based completion and deep learning-based completion. For example, the shape of the occluded part can be restored by interpolation or fitting using the geometric information surrounding the occluded area. Alternatively, a trained deep learning model can be used to predict the possible shape of the occluded part based on existing 3D information. This improves the completeness and accuracy of the 3D reconstruction, helping to more accurately analyze the characteristics of the scrap material, such as obtaining more accurate results when calculating volume. Ultimately, through this series of steps, a complete three-dimensional model of the leftover materials in the cattle and sheep farm environment can be obtained.

[0076] Obtaining the image of the leftover material to be segmented is the starting point of the entire processing flow. Assume there is an image of leftover material from a cattle and sheep farm, containing stacks of leftover material with various colors and textures. First, the image is preprocessed, including denoising and enhancement operations. Denoising can be achieved through median filtering or Gaussian filtering to eliminate random noise and improve image clarity. Enhancement operations can be performed through histogram equalization or contrast stretching to improve image contrast, making the color and texture features of the leftover material more prominent. Next, based on the color and texture features of the leftover material image, the starting point and growth criteria of the region growing algorithm are adaptively selected. Assume there is a piece of red leftover material in the image with a relatively uniform color distribution and smooth texture. The starting point can be selected by calculating the color histogram and texture features of each pixel in the image (such as Local Binary Pattern, LBP), choosing pixels with higher color histogram peaks and more consistent texture features. The growth criteria can be set as a weighted combination of color similarity and texture similarity; for example, a color similarity threshold of 0.1 and a texture similarity threshold of 0.2. By analyzing the color distribution and texture changes in the waste material area, the optimal starting point and growth conditions are determined. Assuming that in the red waste material area, the Euclidean distance between the color feature vector of a pixel and the average color feature vector of the red waste material is less than 0.1, and the Euclidean distance between its texture feature vector and the average texture feature vector of the red waste material is less than 0.2, then this pixel is selected as the starting point. The growth conditions can be dynamically adjusted based on the color and texture distribution of pixels within the area. For example, when the color variance of pixels within the area is large, the color similarity threshold can be appropriately lowered to improve the robustness of growth. Starting from the selected starting point, region growth is performed according to the growth criteria. Assuming the starting point is a pixel in the red waste material, adjacent similar pixels are gradually merged into the same area based on the set color and texture similarity thresholds. As region growth progresses, the color and texture features of pixels within the area will change, requiring dynamic adjustment of the growth threshold. For example, when the proportion of red pixels in the area exceeds 80%, the color similarity threshold can be appropriately increased to avoid mistakenly merging non-red pixels. During region growing, an adaptive threshold control strategy is employed, dynamically adjusting the growth threshold based on the color and texture distribution of pixels within the region. Assuming an initial color similarity threshold of 0.1, as the region grows and the proportion of red pixels gradually increases, the color similarity threshold can be progressively raised to 0.15 to ensure growth accuracy. The texture similarity threshold can also be dynamically adjusted based on changes in texture features within the region to improve segmentation precision. Post-processing of the grown regions, including region merging and hole filling, eliminates over-segmentation and under-segmentation. If multiple small red residue regions are obtained after region growing, regions with high similarity can be merged into a larger red residue region by calculating their color and texture similarity.For holes within the region, morphological operations (such as closing operations) can be used to fill them, resulting in a complete residual material area. The contour boundaries of the segmented residual material area are extracted, and its geometric features, such as area, perimeter, and shape, are calculated. Assuming the area of ​​the segmented red residual material area is 100 square centimeters, the perimeter is 50 centimeters, and the shape factor (the ratio of the squared perimeter to the area) is 0.25, these geometric features provide basic data for subsequent residual material identification and parameter measurement. The segmentation results are fused with the original image to generate a color-coded map, visually displaying the location and extent of the residual material area in the original image. Assuming the original image is an RGB image, the segmented red residual material area can be marked in green. In the generated color-coded map, the red residual material area is covered by green, facilitating manual inspection and verification of segmentation quality. These steps not only improve the segmentation accuracy of the residual material image but also provide reliable data support for subsequent residual material identification and parameter measurement. Adaptively selecting the starting point and growth criteria, as well as dynamically adjusting the growth threshold, effectively handles complex color and texture variations in the residual material image, improving the robustness of the segmentation. The post-processing operation further optimized the segmentation results, ensuring the integrity and accuracy of the remaining material area.

[0077] Obtaining a dataset of scrap material images is the starting point of the entire processing flow. Assume we have a dataset of scrap material images containing scrap materials of different materials, colors, and shapes. First, the images are preprocessed, including denoising and normalization operations. Denoising can be achieved through median filtering, eliminating random noise in the image and improving image clarity. Normalization adjusts the pixel values ​​of the image to the same scale, for example, normalizing all pixel values ​​to between 0 and 1, to ensure consistency in subsequent feature extraction. For the standardized scrap material images, color histogram features, Gabor texture features, and Hu invariant moment features are extracted to construct the feature vector of the scrap material image. Assuming we have a scrap material image, the color histogram feature can reflect the distribution of different colors in the image, for example, red pixels account for 30%, blue pixels account for 20%, etc. The Gabor texture feature extracts the texture information of the image through Gabor filters of multiple directions and scales. Assume that at a certain direction and scale, the texture feature vector of the scrap material is represented as [0.5, 0.3, 0.2]. The Hu invariant moment feature is used to describe the geometry of the image. Assume the Hu invariant moment feature vector of the scrap image is [0.1, 0.2, 0.3]. These three feature vectors are combined into a single comprehensive feature vector, for example, [0.3, 0.2, 0.5, 0.3, 0.2, 0.1, 0.2, 0.3]. Based on the feature vectors of the scrap image, the similarity between scrap samples is calculated using Euclidean distance. Assume there are two scrap samples with feature vectors [0.3, 0.2, 0.5, 0.3, 0.2, 0.1, 0.2, 0.3] and [0.4, 0.3, 0.6, 0.2, 0.1, 0.2, 0.3, 0.4]. Calculating the Euclidean distance between these two vectors yields a similarity value of 0.5. A similarity matrix of the scrap samples is then constructed, arranging the similarity values ​​of each sample with all other samples into a matrix. Assuming there are 10 samples, the similarity matrix is ​​a 10x10 matrix, where each element represents the similarity value between corresponding samples. The similarity matrix is ​​used as input to a spectral clustering algorithm. By solving for the eigenvalues ​​and eigenvectors of the Laplacian matrix, the waste material samples are mapped to a low-dimensional space. Assuming the spectral clustering algorithm maps the samples to a two-dimensional space, the coordinates of each sample in that space are obtained. In the low-dimensional space, the K-means clustering algorithm is used to cluster the waste material samples. Assuming an initial set of 3 cluster centers, through iterative calculation, samples are assigned to the nearest cluster centers, ultimately forming 3 clusters. The optimal number of clusters is determined using the maximum modularity criterion. Assuming a maximum modularity value of 0.8 corresponds to 3 clusters, the optimal number of clusters is determined to be 3. Based on the clustering results, image sets of different types of waste materials are obtained.

[0078] Next, the Canny edge detection algorithm is used to extract edges from the preprocessed image. The Canny algorithm, through non-maximum suppression and double thresholding, can accurately detect image edges. For example, for a Gaussian-filtered image of leftover material, the Canny algorithm can clearly outline the edge contour of the leftover material, generating a binary edge image where white lines represent the edges. Contour analysis is then performed on the extracted leftover material edge contour image, and the polygonal features of the contour are obtained through a contour approximation algorithm. Contour approximation algorithms, such as the Douglas-Peucker algorithm, can simplify the shape representation of the contour and reduce the amount of data. For example, a complex, irregular leftover material contour can be simplified into a polygon with several vertices through contour approximation, greatly simplifying subsequent geometric attribute calculations. Based on the polygonal features of the leftover material contour, the geometric attributes of the polygon are calculated, including shape feature parameters such as the number of sides, side lengths, and interior angles. For example, a simplified polygonal outline has five sides, each with lengths of 10cm, 12cm, 8cm, 11cm, and 9cm, and interior angles of 90°, 80°, 100°, 85°, and 95°. These geometric properties provide the foundational data for subsequent dimensional calculations. Based on the polygon's geometric properties, the length, width, and other dimensional information of the leftover material are further calculated using the minimum bounding rectangle or minimum bounding circle method. For example, for the polygonal outline mentioned above, the approximate dimensions of the leftover material can be obtained by calculating the length and width of its minimum bounding rectangle. Assuming the minimum bounding rectangle has a length of 15cm and a width of 10cm, this dimensional information provides an important basis for classifying the leftover material. Statistical analysis is then performed on the calculated leftover material dimensions to calculate statistical characteristics such as the average length, width, and area of ​​the leftover material. For example, by statistically analyzing the dimensional information of a batch of leftover material samples, the average length is found to be 14cm, the average width 9cm, and the average area 126cm². 2 These statistical characteristics can reflect the overall distribution of surplus materials and provide a basis for classification.

[0079] The process begins by acquiring an image of the waste material to be inspected. First, it is converted to a grayscale image. This step simplifies image processing and reduces computational complexity. Grayscale images contain only grayscale information, eliminating interference from color information and making subsequent image analysis more efficient. For example, after grayscale conversion, the complex color information of a color waste material image is transformed into grayscale values ​​between 0 and 255, facilitating subsequent processing. Next, a grayscale histogram is calculated. A histogram is a statistical representation of the grayscale distribution of an image, visually showing the distribution of the number of pixels at different grayscale levels. The peaks and valleys in the histogram are then identified. Peaks represent areas where grayscale values ​​occur frequently, while valleys represent areas where they occur less frequently. For example, in a grayscale histogram, if two distinct peaks appear at grayscale values ​​of 100 and 150, and a lower valley appears at 125, this indicates two main grayscale regions in the image. Finally, the histogram is checked for multiple peaks; if so, the valleys between adjacent peaks are identified. The presence of multiple peaks usually indicates the existence of multiple distinct regions in the image, potentially due to stains, material variations, or other reasons. For example, if the histogram has a peak at grayscale values ​​of 80 and 180, and a valley at 130, it suggests the presence of two different grayscale regions, with the valley at 130 serving as the boundary between them. The valley value is compared to a preset threshold; if it falls below this threshold, the surface of the remaining material is considered to be stained. This preset threshold, determined empirically or experimentally, distinguishes between normal and stained areas. For instance, if the preset threshold is 120 and the valley value is 110, the area corresponding to that valley value is considered a stain. The stained area is then labeled using a region growing algorithm, a seed-point-based region segmentation algorithm that marks the entire stained area by continuously expanding similar pixels. For example, a pixel with a low grayscale value is selected as the seed point, and the algorithm gradually expands to surrounding pixels with similar grayscale values ​​until the entire stained area is marked. The labeled result is then overlaid onto the original image of the remaining material to obtain the labeled image of the stained area. This step makes the stained area immediately apparent in the original image, facilitating subsequent analysis and processing. For example, in the marked image, the stained area is highlighted and clearly visible. Based on the marked image of the remaining material stain, characteristic parameters such as the location, size, and shape of the stain area are determined. These parameters provide crucial information for subsequent processing. For instance, by measuring the area, perimeter, and geometric center of the stain area, the severity and extent of its impact can be assessed. The characteristic parameters of the stain area are output as the basis for subsequent processing. These parameters help process engineers decide whether to perform cleaning or removal of the remaining material. For example, if the stain area is large and located in a critical area, it may be necessary to remove that portion of the remaining material; if the stain area is small and does not affect usability, cleaning can be performed.

[0080] The algorithm acquires features such as color, texture, shape, and size of scrap material samples to construct scrap material feature vectors. It randomly selects initial cluster centers and calculates the similarity between each scrap material sample and the cluster center. Based on the similarity, the scrap material samples are assigned to the category containing the nearest cluster center. For each category, the cluster centers are recalculated to obtain new cluster centers. The algorithm determines whether the new cluster centers have changed from the previous cluster centers. If the change is less than a preset threshold, the clustering converges, and the algorithm ends. Otherwise, it returns to step 3 to continue iterating. The clustering results are analyzed, and typical scrap material samples from each category are extracted as representatives of that category to construct a clustered scrap material feature database. New scrap material samples are matched with typical samples in the scrap material feature database based on similarity to determine their category, thus achieving rapid classification and management of scrap materials.

[0081] The extracted feature parameters are input into a pre-built Support Vector Machine (SVM) model. SVM is a powerful classification algorithm that separates data into different categories by finding the optimal hyperplane. The assumed SVM model has been trained on a large number of known-category scrap samples and contains multiple support vectors and their corresponding weights. When the feature parameters of the scrap to be identified are input into the model, the model calculates the distance between these parameters and each support vector based on their position in the feature space, thereby predicting its category. By classifying and predicting the input feature parameters using the SVM model, the predicted category label for the scrap is obtained.

[0082] Based on the predicted category label, the corresponding category is retrieved from the pre-clustered waste material feature database. It is assumed that the waste material feature database has already generated multiple categories using a clustering algorithm (such as K-means), each containing feature data from several typical waste material samples. Through retrieval, all waste material feature data under each category is found. If a category matching the predicted category label is found in the waste material feature database, the waste material feature data under that category is extracted. Similarity calculation can employ various methods, such as Euclidean distance and cosine similarity. Assuming Euclidean distance is used, the distance between the waste material to be identified and a certain typical sample is found to be the smallest, indicating the highest similarity. The waste material type with the highest similarity is determined as the final identification result for the waste material to be identified, and this waste material type is output.

[0083] By using a pre-trained neural network model and the residual material size data of the input layer neurons, the weight calculation of the hidden layer neurons yields the inlet size parameters of the collection device corresponding to the output layer.

[0084] Obtain a set of pre-collected scrap size data and the corresponding optimal collection device inlet size parameter as the training dataset for the neural network model. Construct a neural network model containing an input layer, at least one hidden layer, and an output layer. The number of neurons in the input layer is the same as the dimension of the scrap size data, and the number of neurons in the output layer is the same as the dimension of the collection device inlet size parameter. Train the neural network model using the training dataset, adjusting the connection weights between neurons in each layer through backpropagation to enable the model to fit the nonlinear mapping relationship between the scrap size and the optimal inlet size. Test the trained neural network model using a subset of independent scrap size data as input, and determine whether the model's output collection device inlet size parameter is close to the actual optimal value. If the error exceeds a preset threshold, continue training until the requirements are met. Deploy the trained and optimized neural network model to the production environment. When new scrap size data is input, the model can quickly calculate the corresponding optimal collection device inlet size parameter. Based on the output of the neural network model, control the actuator of the collection device to adjust the inlet size to match the calculated optimal parameter value, thereby achieving adaptive optimization. During the production process, data on the size of the remaining material and the inlet size before and after adjustment are continuously collected, and the neural network model is retrained regularly to enable it to adapt to changes in raw material characteristics or process conditions and maintain optimal control performance.

[0085] Information such as the type of waste material, the area of ​​soiling, size parameters, and inlet size parameters is acquired and used as input data for the neural network model. This input data is fed into the pre-trained neural network model, and the predicted trajectory of the collection device is obtained through forward propagation. Based on the predicted trajectory, the specific motion control command sequence for the collection device is determined. This sequence is transmitted to the control unit of the collection device, controlling it to perform waste material collection according to the specified trajectory. During the collection process, sensors collect real-time motion status data and waste material collection information. The collected real-time data is compared with the predicted trajectory, and the deviation is calculated. If the deviation exceeds a preset threshold, the parameters of the neural network model are fine-tuned. The actual motion data and collection results collected during the process are used to incrementally train the neural network model, continuously improving its prediction accuracy and adaptability.

[0086] This embodiment also provides an image recognition-based system for collecting leftover feed in cattle and sheep farms, including:

[0087] The image completion module is used to acquire multi-view, multi-spectral images of leftover materials in the cattle and sheep farm environment. It uses 3D reconstruction technology to complete the leftover materials in the multi-view, multi-spectral images in 3D to obtain a completed image.

[0088] The image segmentation module is used to segment the completed image based on a region growing segmentation algorithm to obtain the segmented residual image region;

[0089] The scrap material classification module is used to extract feature parameters of scrap material image regions, classify scrap material according to feature parameters, and obtain image sets of different types of scrap material; the feature parameters include color histogram, Gabor texture features, and Hu invariant moments.

[0090] The size measurement module is used to extract the outline of the leftover material from the image set to obtain the size information of the leftover material.

[0091] The stain detection module is used to analyze the stains on the surface of the waste material and to mark the stained areas using the region growing method to obtain the marked image of the waste material stains.

[0092] The clustering update module is used to construct a scrap material feature database, calculate the similarity between scrap material samples, update the cluster centers through the similarity matrix, and obtain the clustered scrap material feature database.

[0093] The matching module is used to input the feature parameters of the scrap to be identified into the support vector machine model to obtain the predicted label, and then match the predicted label with the category label in the clustered scrap feature database to obtain the scrap type.

[0094] The inlet size adjustment module is used to dynamically adjust the inlet size of the collection device based on the residual material size information;

[0095] The control module is used to input the type of waste material, the stained area, the size information and the inlet size into the pre-trained neural network model. The model predicts and generates motion trajectory control commands for the collection device, which then controls the collection device to perform waste material collection operations.

[0096] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0097] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0098] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0099] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for collecting leftover feed in cattle and sheep farms based on image recognition, characterized in that, Includes the following steps: Multi-view, multi-spectral images of leftover materials in a cattle and sheep farm environment are acquired. The leftover materials in the multi-view, multi-spectral images are then three-dimensionally completed using 3D reconstruction technology to obtain a completed image. The region growing-based segmentation algorithm is used to segment the completed image to obtain the segmented residual material image region. Extract feature parameters from the image region of scrap material, classify the scrap material according to the feature parameters, and obtain image sets of different types of scrap material; the feature parameters include color histogram, Gabor texture features, and Hu invariant moments; Extract the outline of the remaining material from the image set to obtain the size information of the remaining material; The surface stains of the scrap material were analyzed, and the stained areas were marked using the region growing method to obtain the marked stained image of the scrap material. Based on the marked stained image, the location, size, and shape of the stained areas were determined. Construct a feature database of scrap materials, calculate the similarity between scrap material samples, update the cluster centers through the similarity matrix, and obtain the clustered feature database of scrap materials. The feature parameters of the scrap material to be identified are input into the support vector machine model to obtain the predicted label. The predicted label is then matched with the category label in the clustered scrap material feature database to obtain the scrap material type. The size of the collection device inlet is dynamically adjusted based on the size information of the remaining material. The type of waste material, the area of ​​stain, the size information and the size of the inlet are input into a pre-trained neural network model. The model predicts and generates motion trajectory control commands for the collection device, which then controls the collection device to perform waste material collection operations.

2. The method according to claim 1, characterized in that, Three-dimensional completion of residual material in multi-view, multispectral images includes: Multi-view, multispectral images of leftover materials in a cattle and sheep farm environment are acquired. These images are preprocessed to remove noise. A feature extraction algorithm is used to extract texture and color features of the leftover materials from the noise-removed images. Based on the extracted features, a 3D reconstruction algorithm is used to obtain the 3D spatial structure information of the leftover materials. The 3D size features of the leftover materials are extracted from this 3D spatial structure information. By analyzing the 3D spatial relationship of the leftover materials, it is determined whether there is any stacking or occlusion. If so, a 3D completion algorithm is used to reconstruct the 3D areas of the identified occluded materials to obtain complete 3D structural information of the leftover materials.

3. The method according to claim 1, characterized in that, Segmenting the completed image includes: By analyzing the color distribution and texture changes in the waste material area, the starting point and growth conditions of the region growth algorithm are determined. Starting from the selected starting point, region growth is performed according to the growth conditions. Pixels with similar colors and textures are gradually merged into the same region until no further growth is possible. The grown region is then post-processed, including region merging and hole filling, to obtain the complete waste material area.

4. The method according to claim 1, characterized in that, The image set obtained for different types of scrap includes: The feature parameters of the extracted scrap material image regions include color histogram, Gabor texture features, and Hu invariant moments. A similarity matrix between scrap material samples is constructed based on the feature parameter similarity measure. Spectral clustering algorithm is used to classify the scrap materials. The optimal number of cluster categories is determined by the maximum modularity criterion, and image sets of different types of scrap materials are obtained.

5. The method according to claim 1, characterized in that, Obtaining the remaining material size information includes: The Canny edge detection algorithm is used to extract edges from the image to obtain the edge contour image of the scrap material; the polygon features of the scrap material contour are obtained through a contour approximation algorithm; the geometric properties of the polygon are calculated based on the polygon features, including the number of sides, side lengths, and interior angles of the polygon; the size information of the scrap material is calculated based on the geometric properties of the polygon.

6. The method according to claim 1, characterized in that, Analysis of surface stains on leftover materials includes: Convert the image of the scrap material to a grayscale image; calculate the grayscale histogram of the grayscale image and determine the peak and valley values ​​in the histogram; determine whether there are multiple peaks in the histogram, and if so, obtain the valley value between adjacent peaks; compare the valley value with a preset threshold, and if the valley value is lower than the threshold, determine that there are stains on the surface of the scrap material.

7. A system for collecting leftover feed in cattle and sheep farms based on image recognition, characterized in that, include: The image completion module is used to acquire multi-view, multi-spectral images of leftover materials in the cattle and sheep farm environment. It uses 3D reconstruction technology to complete the leftover materials in the multi-view, multi-spectral images in 3D to obtain a completed image. The image segmentation module is used to segment the completed image based on a region growing segmentation algorithm to obtain the segmented residual image region; The scrap material classification module is used to extract feature parameters of scrap material image regions, classify scrap material according to feature parameters, and obtain image sets of different types of scrap material; the feature parameters include color histogram, Gabor texture features, and Hu invariant moments. The size measurement module is used to extract the outline of the leftover material from the image set to obtain the size information of the leftover material. The stain detection module is used to analyze the stains on the surface of the waste material and to mark the stained areas using the region growing method to obtain a marked image of the waste material stains; based on the marked image of the waste material stains, the location, size and shape of the stained areas are determined. The clustering update module is used to construct a scrap material feature database, calculate the similarity between scrap material samples, update the cluster centers through the similarity matrix, and obtain the clustered scrap material feature database. The matching module is used to input the feature parameters of the scrap to be identified into the support vector machine model to obtain the predicted label, and then match the predicted label with the category label in the clustered scrap feature database to obtain the scrap type. The inlet size adjustment module is used to dynamically adjust the inlet size of the collection device based on the residual material size information; The control module is used to input the type of waste material, the stained area, the size information and the inlet size into the pre-trained neural network model. The model predicts and generates motion trajectory control commands for the collection device, which then controls the collection device to perform waste material collection operations.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-6.

Images