Cow and sheep farm excess material collection method and system based on image recognition

Abstract

The invention provides a cattle and sheep farm excess stock collection method and system based on image recognition, and the method comprises the steps: carrying out the image segmentation processing of an obtained excess stock image, employing a segmentation algorithm based on region growth, adaptively selecting a starting point and a growth criterion of region growth according to the color texture features of the excess stock, and obtaining a segmented excess stock image region; for each type of excess material image set, an edge detection algorithm is adopted to extract excess material contours, excess material shape polygon features are obtained through contour approximation, and size information such as length, width and area of excess materials is calculated according to the polygon features; and inputting the excess material type, the stain area, the size parameter and the inlet size parameter into the neural network model, generating a motion track control instruction of the collection device through model prediction, and controlling the collection device to execute excess material collection operation. According to the invention, a complex excess material scene can be accurately processed, and efficient excess material recovery is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of waste material recovery, and in particular relates to a method and system for collecting waste materials in cattle and sheep farms based on image recognition. Background Art

[0002] In the process of collecting waste materials in cattle and sheep farms, the shapes, sizes and materials of waste materials vary greatly, making it difficult for the collection device to adapt to the collection needs of different types of waste materials. If the parameters of the collection device are not set properly, the collection efficiency and quality will be affected. For example, for large waste materials, if the inlet size of the collection device is set too small, the waste materials cannot enter smoothly, resulting in incomplete collection; and for loose and fine waste materials, if the suction force of the collection device is too strong, the waste materials may be sucked into the device, causing the device to be blocked.

[0003] At the same time, due to the complex environment of cattle and sheep farms and the changing lighting conditions, the surface of the leftovers may be stained, making it difficult for the image recognition system to accurately identify the characteristic parameters of the leftovers. In particular, when the leftovers are stacked or blocked by other debris, the information obtained by the image recognition system is incomplete, affecting the recognition accuracy. In addition, due to the different distribution positions and postures of the leftovers, higher requirements are also placed on the posture estimation of the image recognition system.

[0004] Therefore, how to accurately obtain the characteristic parameters of the residual materials such as shape, size and material based on the image recognition results, and dynamically adjust the structure and working parameters of the collection device to achieve adaptive collection of various types of residual materials is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a method and system for collecting waste materials in cattle and sheep farms based on image recognition to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above object, the present invention provides a method for collecting waste materials in cattle and sheep farms based on image recognition, comprising the following steps:

[0007] Acquire multi-view multi-spectral images of residual materials in a cattle and sheep farm environment, and use three-dimensional reconstruction technology to perform three-dimensional completion on the residual materials in the multi-view multi-spectral images to obtain a completed image;

[0008] Segmenting the completed image based on a segmentation algorithm of region growing to obtain a segmented residual material image region;

[0009] Extract characteristic parameters of the residual material image area, classify the residual materials according to the characteristic parameters, and obtain image sets of different types of residual materials; wherein the characteristic parameters include color histogram, Gabor texture feature and Hu invariant moment;

[0010] Extracting the outline of the residual material from the image set to obtain residual material size information;

[0011] Analyze the stains on the surface of the leftover material, and use the region growing method to mark the stain area to obtain the marked leftover material stain image;

[0012] Construct a waste material feature database, calculate the similarity between waste material samples, update the cluster center through the similarity matrix, and obtain the clustered waste material feature database;

[0013] Input the feature parameters of the leftovers to be identified into the support vector machine model to obtain a predicted label, and match the predicted label with the category label in the clustered leftover feature database to obtain the leftover type;

[0014] Dynamically adjust the size of the collection device entrance according to the residual material size information;

[0015] The type of residual material, stain area, size information and entrance size are input into the pre-trained neural network model, and the motion trajectory control instructions of the collection device are generated through model prediction to control the collection device to perform the residual material collection operation.

[0016] Preferably, performing three-dimensional completion on the residual material in the multi-view multi-spectral image includes:

[0017] Acquire multi-view and multi-spectral images of leftovers in a cattle and sheep farm environment; pre-process the acquired multi-view and multi-spectral images to remove image noise; use a feature extraction algorithm to extract texture and color features of the leftovers from the images after removing image noise; use a three-dimensional reconstruction algorithm to obtain the three-dimensional spatial structure information of the leftovers based on the extracted leftover features; extract the three-dimensional size features of the leftovers from the three-dimensional spatial structure information of the leftovers; determine whether the leftovers are stacked or blocked by analyzing the three-dimensional spatial position relationship of the leftovers, and if so, use a three-dimensional completion algorithm to perform three-dimensional reconstruction on the determined blocked leftover area to obtain complete three-dimensional structural information of the leftovers.

[0018] Preferably, segmenting the completed image includes:

[0019] By analyzing the color distribution and texture changes of the waste area, the starting point position and growth conditions of the region growing algorithm are determined; starting from the selected starting point, regional growth is performed according to the growth conditions; pixels of similar colors and textures are gradually merged into the same region until no further growth is possible; post-processing is performed on the grown region, including region merging and hole filling, to obtain a complete waste area.

[0020] Preferably, obtaining an image set of different types of waste materials includes:

[0021] The feature parameters of the residual material image area, including color histogram, Gabor texture features and Hu invariant moments, are extracted. The similarity matrix between the residual material samples is constructed according to the similarity measurement of the feature parameters. The spectral clustering algorithm is used to classify the residual materials. The optimal number of clustering categories is determined by the maximum modularity criterion, and the image collection of different types of residual materials is obtained.

[0022] Preferably, obtaining the residual material size information includes:

[0023] The Canny edge detection algorithm is used to extract the edge of the image to obtain the edge contour image of the residual material; the polygonal features of the residual material contour are obtained through the contour approximation algorithm; according to the polygonal features, the geometric properties of the polygon are calculated, wherein the geometric properties include the number of sides, side length and internal angle of the polygon; and the size information of the residual material is calculated based on the geometric properties of the polygon.

[0024] Preferably, analyzing the stains on the surface of the residual material includes:

[0025] Convert the residual material image into a grayscale image; calculate the grayscale histogram of the grayscale image and determine the peak and valley values ​​in the histogram; determine whether there are multiple peaks in the histogram, and if so, obtain the valley value between adjacent peaks; compare the valley value with a preset threshold, and if the valley value is lower than the threshold, determine that there are stains on the surface of the residual material.

[0026] The present invention also provides a cattle and sheep farm waste material collection system based on image recognition, comprising:

[0027] An image completion module is used to obtain multi-view multi-spectral images of residual materials in a cattle and sheep farm environment, and to perform three-dimensional completion on the residual materials in the multi-view multi-spectral images through three-dimensional reconstruction technology to obtain a completed image;

[0028] An image segmentation module, used for segmenting the completed image based on a region growing segmentation algorithm to obtain a segmented residual image region;

[0029] The residual material classification module is used to extract the characteristic parameters of the residual material image area, classify the residual materials according to the characteristic parameters, and obtain a set of images of different types of residual materials; wherein the characteristic parameters include color histogram, Gabor texture feature and Hu invariant moment;

[0030] A size measurement module, used for extracting the outline of the residual material from the image set to obtain the residual material size information;

[0031] The stain detection module is used to analyze the stain situation on the surface of the residual material and mark the stain area using the region growing method to obtain the marked residual material stain image;

[0032] The clustering update module is used to build a residual material feature database, calculate the similarity between residual material samples, update the cluster center through the similarity matrix, and obtain the residual material feature database after clustering;

[0033] A matching module is used to input the feature parameters of the leftover material to be identified into the support vector machine model to obtain a predicted label, and match the predicted label with the category label in the clustered leftover material feature database to obtain the leftover material type;

[0034] An inlet size adjustment module is used to dynamically adjust the size of the collection device inlet according to the residual material size information;

[0035] The control module is used to input the residual material type, stain area, size information and entrance size into the pre-trained neural network model, generate the motion trajectory control instructions of the collection device through model prediction, and control the collection device to perform the residual material collection operation.

[0036] The present invention provides a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0037] The present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method are implemented.

[0038] The present invention provides a computer program product, comprising a computer program, which implements the steps of the method when executed by a processor.

[0039] Compared with the prior art, the present invention has the following advantages and technical effects:

[0040] The present invention provides a method and system for collecting waste materials in cattle and sheep farms based on image recognition. For complex scenes where waste materials are stacked and blocked, multi-view multi-spectral imaging and three-dimensional reconstruction technology are used to obtain spatial structural information of waste materials, so as to achieve three-dimensional completion of blocked waste materials. Through image segmentation, feature extraction and cluster analysis, a waste material feature database is established to realize the identification and classification of different types of waste materials. Combined with stain detection and size measurement, the size of the entrance of the collection device is dynamically adjusted, and an optimal collection trajectory is generated. The present invention integrates computer vision, machine learning and other technologies, solves the problems of difficult identification, classification and collection in the collection of waste materials in cattle and sheep farms, improves the intelligence and automation level of waste material collection, and has important practical value. The present invention can accurately handle complex waste material scenes and realize efficient waste material recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Embodiment 1

[0046] like Figure 1 As shown, this embodiment provides a method for collecting waste materials in cattle and sheep farms based on image recognition, comprising the following steps:

[0047] Acquire multi-view multi-spectral images of residual materials in a cattle and sheep farm environment, and use three-dimensional reconstruction technology to perform three-dimensional completion on the residual materials in the multi-view multi-spectral images to obtain a completed image;

[0048] The completed image is segmented based on a segmentation algorithm of region growing to obtain the segmented residual image region;

[0049] Extract characteristic parameters of the residual material image area, classify the residual materials according to the characteristic parameters, and obtain image sets of different types of residual materials; wherein the characteristic parameters include color histogram, Gabor texture feature and Hu invariant moment;

[0050] Extract the residual material contour from the image set to obtain the residual material size information;

[0051] Analyze the stains on the surface of the leftover material, and use the region growing method to mark the stain area to obtain the marked leftover material stain image;

[0052] Construct a waste material feature database, calculate the similarity between waste material samples, update the cluster center through the similarity matrix, and obtain the clustered waste material feature database;

[0053] Input the feature parameters of the leftovers to be identified into the support vector machine model to obtain a predicted label, and match the predicted label with the category label in the clustered leftover feature database to obtain the leftover type;

[0054] Dynamically adjust the size of the collection device entrance according to the residual material size information;

[0055] The type of residual material, stain area, size information and entrance size are input into the pre-trained neural network model, and the motion trajectory control instructions of the collection device are generated through model prediction to control the collection device to perform the residual material collection operation.

[0056] The specific steps include:

[0057] S101. Obtain multi-view and multi-spectral images of the residual materials in the cattle and sheep farm environment. In view of the stacking and occlusion of the residual materials, use three-dimensional reconstruction technology to obtain the spatial structure information of the residual materials, extract the three-dimensional size characteristics of the residual materials, judge the stacking of the residual materials by analyzing the three-dimensional spatial position relationship of the residual materials, and complete the occluded residual materials in three dimensions.

[0058] Specifically, multi-view and multi-spectral images of leftovers in a cattle and sheep farm environment are obtained; the obtained multi-view and multi-spectral images are preprocessed to remove image noise and improve image quality; a feature extraction algorithm is used to extract texture, color and other features of the leftovers from the multi-view and multi-spectral images; based on the extracted leftover features, a three-dimensional reconstruction algorithm is used to obtain the three-dimensional spatial structure information of the leftovers; the three-dimensional size features of the leftovers are extracted from the three-dimensional spatial structure information of the leftovers; by analyzing the three-dimensional spatial position relationship of the leftovers, it is determined whether the leftovers are stacked or occluded, and if so, the occluded leftover area is determined; for the determined occluded leftover area, a three-dimensional completion algorithm is used to perform three-dimensional reconstruction to obtain complete three-dimensional structure information of the leftovers.

[0059] S102, performing image segmentation processing on the acquired residual material image, using a segmentation algorithm based on region growing, adaptively selecting a starting point and a growth criterion for region growing according to the residual material color and texture features, and obtaining a segmented residual material image region.

[0060] Specifically, the starting point and growth criteria of the region growing algorithm are adaptively selected according to the color and texture characteristics of the residual material image. The optimal starting point position and growth conditions are determined by analyzing the color distribution and texture changes of the residual material area. Starting from the selected starting point, regional growth is performed according to the growth criteria. Pixels of similar color and texture are gradually merged into the same area until no further growth is possible. In the process of region growing, an adaptive threshold control strategy is adopted to dynamically adjust the growth threshold according to the color and texture distribution of the pixels in the region to improve the segmentation accuracy and robustness. The grown area is post-processed, including region merging, hole filling, etc., to eliminate over-segmentation and under-segmentation phenomena and obtain a complete residual material area.

[0061] S103, extracting characteristic parameters such as color histogram, Gabor texture features and Hu invariant moments of the residual material image area, constructing a similarity matrix between residual material samples according to the characteristic parameter similarity measurement, adopting a spectral clustering algorithm to classify the residual materials, determining the optimal number of clustering categories through a maximum modularity criterion, and obtaining an image set of different types of residual materials.

[0062] Specifically, for the leftover image, the color histogram features, Gabor texture features and Hu invariant moment features are extracted to construct the feature vector of the leftover image. According to the feature vector of the leftover image, the Euclidean distance metric is used to calculate the similarity between the leftover samples, and the similarity matrix of the leftover samples is constructed. The similarity matrix is ​​used as the input of the spectral clustering algorithm, and the leftover samples are mapped to the low-dimensional space by solving the eigenvalues ​​and eigenvectors of the Laplace matrix. In the low-dimensional space, the K-means clustering algorithm is used to cluster the leftover samples, and the optimal number of clustering categories is determined by the maximum modularity criterion to obtain the clustering results of the leftovers. According to the clustering results, image sets of different types of leftovers are obtained, and the leftover images of each category are visualized and analyzed. For different types of leftover image sets, convolutional neural networks are used for feature extraction and representation learning to construct the deep features of the leftover images for subsequent leftover recognition and matching tasks.

[0063] S104, for each type of waste material image set, using the Canny edge detection algorithm to extract the waste material contour, obtaining the waste material shape polygonal features through contour approximation, and calculating the length, width, area and other size information of the waste material according to the polygonal features.

[0064] Specifically, the inputted scrap image is preprocessed, and the image is smoothed and denoised by using a Gaussian filter algorithm to reduce noise interference in the image. According to the preprocessed scrap image, the Canny edge detection algorithm is used to extract the edge of the image to obtain the edge contour image of the scrap. The extracted scrap edge contour image is subjected to contour analysis, and the polygonal features of the scrap contour are obtained by a contour approximation algorithm to simplify the shape representation of the contour. According to the polygonal features of the scrap contour, the geometric properties of the polygon are calculated, including shape feature parameters such as the number of edges, edge length, and internal angle of the polygon. Based on the geometric properties of the polygon, the length, width and other size information of the scrap are further calculated, and the size is obtained by the method of the minimum circumscribed rectangle or the minimum circumscribed circle. The calculated scrap size information is statistically analyzed, and the statistical features such as the average length, width and area of ​​the scrap are calculated as the basis for the classification of the scrap. The support vector machine (SVM) algorithm is used to classify the scrap, and the SVM classifier is trained according to the size statistical features of the scrap to realize the automatic classification and identification of the scrap.

[0065] S105, analyzing the peak-valley features of the grayscale histogram of the residual material image. If there are multiple peaks in the histogram and the valley values ​​between the peaks are lower than a preset threshold, it is determined that there are stains on the residual material surface, and the stain area is marked using a region growing method to obtain a marked residual material stain image.

[0066] Specifically, the image of the residual material to be detected is obtained and converted into a grayscale image; the grayscale histogram of the grayscale image is calculated to determine the peak and valley values ​​in the histogram; it is determined whether there are multiple peaks in the histogram, and if so, the valley value between adjacent peaks is obtained; the valley value is compared with a preset threshold value, and if the valley value is lower than the threshold value, it is determined that there are stains on the surface of the residual material; the stain area is marked by using the region growing method, and the marking result is superimposed on the original residual material image to obtain a marked residual material stain image; according to the marked residual material stain image, characteristic parameters such as the position, size and shape of the stain area are determined; the characteristic parameters of the stain area are output as a basis for subsequent process processing, so as to perform operations such as cleaning or removing the residual material.

[0067] S106, constructing a waste material feature database, using cluster centers to select waste material samples as cluster centers, calculating the similarity between waste material samples based on features such as color, texture, shape and size of the waste materials, updating the cluster centers through a similarity matrix, and continuously iterating until the cluster centers no longer change, thereby obtaining a clustered waste material feature database.

[0068] Specifically, the features of the waste material samples such as color, texture, shape and size are obtained to construct a waste material feature vector; the initial cluster center is randomly selected, and the similarity between each waste material sample and the cluster center is calculated; the waste material samples are divided into the category where the nearest cluster center is located according to the similarity; the cluster center is recalculated for the waste material samples in each category to obtain a new cluster center; it is determined whether the new cluster center has changed from the cluster center of the previous round, if the change is less than a preset threshold, the clustering converges and the algorithm ends; otherwise, it returns to step 3 to continue iterating; the clustering results are analyzed, and the typical waste material samples of each category are extracted as representatives of the category, and a clustered waste material feature database is constructed; the new waste material samples are matched with the typical samples in the waste material feature database for similarity, and the category to which they belong is determined, so as to realize the rapid classification and management of waste materials.

[0069] S107, inputting the feature parameters of the leftover material to be identified into the support vector machine model, obtaining the category label of the leftover material through model prediction, matching the predicted label with the category label in the clustered leftover material feature database, and finding the most similar leftover material type.

[0070] Specifically, characteristic parameters of the leftover material to be identified are obtained, and the characteristic parameters are input into a pre-built support vector machine model; the input characteristic parameters are classified and predicted by the support vector machine model to obtain a predicted category label of the leftover material; according to the predicted category label, the corresponding category is retrieved in a leftover material feature database generated by clustering in advance; if a category matching the predicted category label is found in the leftover material feature database, the leftover material feature data under the category is extracted; the similarity between the characteristic parameters of the leftover material to be identified and the extracted leftover material feature data is calculated, and the leftover material type with the highest similarity is selected; the leftover material type with the highest similarity is determined as the final identification result of the leftover material to be identified, and the leftover material type is output; if a category matching the predicted category label is not found in the leftover material feature database, it is determined that the identification has failed, and the user is prompted to obtain the characteristic parameters of the leftover material to be identified again.

[0071] S108. Dynamically adjust the entrance size of the collection device according to the residual material size information, establish a neural network mapping model between the residual material size parameters and the entrance size, use the residual material length, width and area as neuron inputs, calculate the entrance size parameters through the weight calculation of the hidden layer neurons, and optimize the parameters on the training samples.

[0072] Specifically, the size information of the residual material is obtained, including the length, width and area of ​​the residual material; according to the obtained residual material size information, the input layer neurons of the neural network model are determined, which correspond to the length, width and area of ​​the residual material respectively; through the pre-trained neural network model, the residual material size data of the input layer neurons are used, and the weight calculation of the hidden layer neurons is performed to obtain the collection device entrance size parameters corresponding to the output layer; it is judged whether the calculated collection device entrance size parameters meet the preset optimization conditions, if so, the parameters are used as the optimal entrance size, if not, return to the third step to continue iterative optimization; according to the determined optimal entrance size parameters, the entrance adjustment mechanism of the collection device is controlled to dynamically adjust the entrance size to match the residual material size; a photoelectric sensor is set at the entrance of the collection device to detect the time and position of the residual material entering, and transmit the detection signal to the control unit; the control unit determines whether the residual material enters the collection device smoothly according to the received residual material entry signal, and if abnormal conditions such as jamming or falling occur, a warning signal is issued to notify the staff to handle it.

[0073] S109, inputting the residual material type, stain area, size parameters and inlet size parameters into the neural network model, generating motion trajectory control instructions for the collection device through model prediction, and controlling the collection device to perform the residual material collection operation.

[0074] Specifically, information such as the type of residual material, stain area, size parameters, and inlet size parameters is obtained as input data for the neural network model. The input data is passed into the pre-trained neural network model, and the prediction result of the motion trajectory of the collection device is obtained through forward propagation calculation. According to the motion trajectory result predicted and generated by the neural network model, the specific motion control instruction sequence of the collection device is determined. The generated control instruction sequence is transmitted to the control unit of the collection device to control the collection device to perform the residual material collection operation according to the specified motion trajectory. During the collection operation of the collection device, the motion state data of the device and the residual material collection situation information are collected in real time through sensors. The collected real-time state data is compared with the predicted motion trajectory, and the deviation value is calculated. If the deviation exceeds the preset threshold, the parameter fine-tuning of the neural network model is triggered. The actual motion data and collection results collected during the collection process are used to perform incremental training on the neural network model to continuously improve the prediction accuracy and adaptability of the model.

[0075] In this embodiment, a multi-view multi-spectral image of the leftovers in the cattle and sheep farm environment is obtained. This step is to capture the information of the leftovers comprehensively. Multi-view refers to shooting from different angles, which can avoid omissions caused by occlusion or a single view. Multi-spectral can provide information beyond the visible light range, such as infrared and ultraviolet bands, which helps to analyze the material and composition of the leftovers. For example, four cameras can be set up, located in the four corners of the cattle and sheep farm, and each camera is equipped with multiple filters to obtain images of visible light, near-infrared and ultraviolet bands respectively. In this way, the reflection characteristics of the leftovers under different spectra can be obtained, which helps to distinguish different materials such as feed residues and plastic films. The obtained multi-view multi-spectral images are preprocessed to remove image noise and improve image quality. The image may be affected by factors such as illumination changes and sensor noise during the acquisition process, resulting in quality degradation. Preprocessing includes steps such as filtering denoising and illumination correction, aiming to improve the clarity and contrast of the image. For example, Gaussian filtering is used to remove random noise in the image, and histogram equalization is used to enhance the contrast of the image, so that the edges and textures of the leftovers are clearer. These preprocessing techniques can effectively improve image quality and lay a good foundation for subsequent feature extraction and three-dimensional reconstruction. Feature extraction algorithms are used to extract texture, color and other features of the residual material from multi-view multi-spectral images. These features are the key to identifying and analyzing residual materials. Texture features describe the surface structure of the image, such as roughness and directionality; color features reflect the reflective properties of the residual material. Gray-level co-occurrence matrices can be used to extract texture features of the residual material, such as energy and contrast, which can reflect the smoothness and uniformity of the residual material surface. Color features can be represented by color histograms or color moments, such as calculating the color mean and variance of the residual material in each spectral band to distinguish different types and states of residual materials. Based on the extracted residual material features, a three-dimensional reconstruction algorithm is used to obtain the three-dimensional spatial structure information of the residual material. The purpose of this step is to restore the three-dimensional shape of the residual material from the two-dimensional image. Common three-dimensional reconstruction algorithms include stereo vision, structured light, and motion-based reconstruction. For example, using stereo vision technology, by matching corresponding points in images of different viewpoints, the three-dimensional coordinates of these points are calculated, and then a three-dimensional point cloud of the residual material is generated. Point cloud data can intuitively display the spatial form of the leftovers and provide data support for subsequent analysis. The three-dimensional size features of the leftovers are extracted from the three-dimensional spatial structure information of the leftovers. Three-dimensional size features are important parameters that describe the geometric shape of the leftovers, including length, width, height, volume, etc. These features can be calculated directly from the three-dimensional model or point cloud data of the leftovers. For example, the length, width, and height of the leftovers can be obtained by calculating the minimum outer bounding box of the point cloud data. The point cloud is voxelized, the number of voxels is counted, and then the volume of the leftovers is calculated. These size features can be used to evaluate the quantity and distribution of leftovers and assist farm management. By analyzing the three-dimensional spatial position relationship of the leftovers, it is determined whether there is stacking occlusion of the leftovers. If so, the occluded leftover area is determined.Since the leftovers are often piled together, there will be occlusion, which will affect the acquisition of complete information of the leftovers. By analyzing the relative position relationship between the leftover point clouds, the occluded area can be identified. For example, if a part of a leftover point cloud is occluded by another leftover point cloud, the density of the part of the point cloud will be significantly lower than that of other parts, and its edge contour may be incomplete. Using these features, the occluded area can be detected. For the determined occluded leftover area, a 3D completion algorithm is used to perform 3D reconstruction to obtain complete 3D structural information of the leftovers. The purpose of completion is to restore the 3D structure of the occluded area so that the 3D model of the leftovers is more complete. Commonly used 3D completion methods include completion based on geometric rules and completion based on deep learning. For example, the geometric information around the occluded area can be used to restore the shape of the occluded part by interpolation or fitting. Or a trained deep learning model can be used to predict the possible shape of the occluded part based on the existing 3D information. This can improve the completeness and accuracy of 3D reconstruction and help to more accurately analyze the characteristics of the leftovers, such as obtaining more accurate results when calculating the volume. Ultimately, through this series of steps, a complete three-dimensional model of the waste in the cattle and sheep farm environment can be obtained.

[0076] Obtaining the image of the leftovers to be segmented is the starting point of the entire processing flow. Suppose there is an image of leftovers from a cattle and sheep farm, in which leftovers of various colors and textures are stacked together. First, the image is preprocessed, including denoising and enhancement operations. Denoising can be achieved through median filtering or Gaussian filtering to eliminate random noise in the image and improve the clarity of the image. The enhancement operation can improve the contrast of the image through histogram equalization or contrast stretching, making the color and texture features of the leftovers more obvious. Next, according to the color and texture features of the leftover image, the starting point and growth criterion of the region growing algorithm are adaptively selected. Suppose there is a piece of red leftovers in the image, whose color distribution is relatively uniform and the texture is relatively smooth. The color histogram and texture features (such as local binary pattern LBP) of each pixel in the image can be calculated, and the pixel with a higher color histogram peak and more consistent texture features can be selected as the starting point. The growth criterion can be set as a weighted combination of color similarity and texture similarity, for example, the color similarity threshold is set to 0.1 and the texture similarity threshold is set to 0.2. By analyzing the color distribution and texture changes of the residual material area, the optimal starting point position and growth conditions are determined. Assuming that in the red residual material area, the Euclidean distance between the color feature vector of a certain pixel and the average color feature vector of the red residual material is less than 0.1, and the Euclidean distance between its texture feature vector and the average texture feature vector of the red residual material is less than 0.2, then the pixel is selected as the starting point. The growth conditions can be dynamically adjusted according to the color and texture distribution of the pixels in the region. For example, when the color variance of the pixels in the region is large, the color similarity threshold is appropriately reduced to improve the robustness of the growth. Starting from the selected starting point, regional growth is performed according to the growth criterion. Assuming that the starting point is a pixel in the red residual material, adjacent similar pixels are gradually merged into the same region according to the set color and texture similarity thresholds. As the region grows, the color and texture characteristics of the pixels in the region will change, and the growth threshold needs to be dynamically adjusted at this time. For example, when the proportion of red pixels in the region exceeds 80%, the color similarity threshold can be appropriately increased to avoid mistakenly merging non-red pixels. In the process of region growing, an adaptive threshold control strategy is adopted to dynamically adjust the growth threshold according to the color and texture distribution of pixels in the region. Assuming that the initial color similarity threshold is 0.1, as the region grows, the proportion of red pixels in the region gradually increases, and the color similarity threshold can be gradually increased to 0.15 to ensure the accuracy of growth. The texture similarity threshold can also be dynamically adjusted according to the changes in texture features in the region to improve segmentation accuracy. The grown regions are post-processed, including region merging and hole filling, to eliminate over-segmentation and under-segmentation. Assuming that multiple small red residue regions are obtained after region growing, the color and texture similarity of these regions can be calculated to merge the regions with higher similarity into a large red residue region.For the holes in the region, morphological operations (such as closing operations) can be used to fill them and obtain a complete residual material region. The contour boundary of the residual material region obtained by segmentation is extracted, and the geometric features of the region, such as area, perimeter, shape, etc., are calculated. Assume that the area of ​​the red residual material region obtained by segmentation is 100 square centimeters, the perimeter is 50 centimeters, and the shape factor (the ratio of the square of the perimeter to the area) is 0.25. These geometric features can provide basic data for subsequent residual material identification and parameter measurement. The segmentation results are fused with the original image to generate a color marking map to intuitively display the position and range of the residual material region in the original image. Assuming that the original image is an RGB image, the red residual material region obtained by segmentation can be marked with green. In the generated color marking map, the red residual material region is covered with green, which is convenient for manual inspection and verification of the segmentation quality. Through the above steps, not only the segmentation accuracy of the residual material image can be improved, but also reliable data support can be provided for subsequent residual material identification and parameter measurement. Adaptive selection of starting points and growth criteria, as well as dynamic adjustment of growth thresholds, can effectively cope with complex color and texture changes in residual material images and improve the robustness of segmentation. Post-processing operations further optimize the segmentation results and ensure the integrity and accuracy of the residual material area.

[0077] Obtaining a dataset of leftover images is the starting point of the entire processing flow. Assume that there is a dataset of leftover images, which contains leftovers of different materials, colors, and shapes. First, the image is preprocessed, including denoising and normalization operations. Denoising can be achieved through median filtering to eliminate random noise in the image and improve the clarity of the image. The normalization operation adjusts the pixel values ​​of the image to the same scale, for example, normalizing all pixel values ​​to between 0 and 1, so as to ensure the consistency of subsequent feature extraction. For the standardized leftover image, the color histogram feature, Gabor texture feature, and Hu invariant moment feature are extracted to construct the feature vector of the leftover image. Assume that there is a leftover image, the color histogram feature can reflect the distribution of different colors in the image, for example, red pixels account for 30%, blue pixels account for 20%, etc. The Gabor texture feature extracts the texture information of the image through Gabor filters in multiple directions and scales. Assume that in a certain direction and scale, the texture feature vector of the leftover is expressed as [0.5, 0.3, 0.2]. The Hu invariant moment feature is used to describe the geometric shape of the image. Assume that the Hu invariant moment feature vector of the leftover image is [0.1, 0.2, 0.3]. Combine the above three feature vectors into a comprehensive feature vector, for example [0.3, 0.2, 0.5, 0.3, 0.2, 0.1, 0.2, 0.3]. According to the feature vector of the leftover image, the Euclidean distance metric is used to calculate the similarity between the leftover samples. Assume that there are two leftover samples with feature vectors of [0.3, 0.2, 0.5, 0.3, 0.2, 0.1, 0.2, 0.3] and [0.4, 0.3, 0.6, 0.2, 0.1, 0.2, 0.3, 0.4], respectively. By calculating the Euclidean distance between the two vectors, the similarity value is 0.5. Construct the similarity matrix of the leftover samples and arrange the similarity values ​​of each sample with all other samples into a matrix. Assuming there are 10 samples, the similarity matrix is ​​a 10x10 matrix, and each element represents the similarity value between the corresponding samples. The similarity matrix is ​​used as the input of the spectral clustering algorithm, and the residual samples are mapped to the low-dimensional space by solving the eigenvalues ​​and eigenvectors of the Laplace matrix. Assuming that the samples are mapped to the two-dimensional space by the spectral clustering algorithm, the coordinates of each sample in the two-dimensional space are obtained. In the low-dimensional space, the K-means clustering algorithm is used to cluster the residual samples. Assuming that the initial clustering centers are set to 3, the samples are assigned to the nearest clustering centers through iterative calculations, and finally 3 clusters are formed. The optimal number of cluster categories is determined by the maximum modularity criterion. Assuming that the maximum modularity value is 0.8 and the corresponding number of clusters is 3, the optimal number of cluster categories is determined to be 3. According to the clustering results, the image collection of different types of residual materials is obtained.

[0078] Next, the Canny edge detection algorithm is used to extract the edges of the preprocessed image. The Canny algorithm can accurately detect the edges of the image through non-maximum suppression and double threshold processing. For example, for a Gaussian filtered residual material image, the Canny algorithm can clearly outline the edge contour of the residual material and generate a binary edge image, in which the white line represents the edge of the residual material. The extracted residual material edge contour image is subjected to contour analysis, and the polygonal features of the residual material contour are obtained through the contour approximation algorithm. Contour approximation algorithms such as the Douglas-Peucker algorithm can simplify the shape representation of the contour and reduce the amount of data. For example, a complex irregular residual material contour can be simplified into a polygon containing several vertices through the contour approximation algorithm, which greatly simplifies the subsequent geometric attribute calculation. According to the polygonal features of the residual material contour, the geometric properties of the polygon are calculated, including shape feature parameters such as the number of edges, edge length, and internal angle of the polygon. For example, a simplified polygonal outline has 5 sides, and the length of each side is 10cm, 12cm, 8cm, 11cm and 9cm, and the internal angles are 90°, 80°, 100°, 85° and 95°, respectively. These geometric properties provide basic data for the subsequent calculation of size information. Based on the geometric properties of the polygon, the length, width and other size information of the residual material are further calculated, and the size is obtained by the method of the minimum circumscribed rectangle or the minimum circumscribed circle. For example, for the above polygonal outline, the approximate size of the residual material can be obtained by calculating the length and width of its minimum circumscribed rectangle. Assuming that the length of the minimum circumscribed rectangle is 15cm and the width is 10cm, these size information provides an important basis for the classification of the residual material. The calculated residual material size information is statistically analyzed to calculate the statistical characteristics of the residual material such as the average length, width and area. For example, by statistically analyzing the size information of a batch of residual material samples, the average length is 14cm, the average width is 9cm, and the average area is 126cm 2 These statistical characteristics can reflect the overall distribution of the waste material and provide a basis for classification.

[0079] Obtain the image of the residual material to be detected and first convert it into a grayscale image. This step is to simplify the image processing process and reduce the computational complexity. Grayscale images only contain grayscale information, eliminating the interference of color information, making subsequent image analysis more efficient. For example, after a color residual material image is converted into grayscale values ​​between 0 and 255, it is convenient for subsequent processing. Calculate the grayscale histogram of the grayscale image. The histogram is a statistical representation of the grayscale distribution of the image. The distribution of the number of pixels of different grayscale levels in the image can be intuitively seen through the histogram. Determine the peaks and valleys in the histogram. The peaks represent the areas with higher grayscale values ​​in the image, while the valleys are the areas with lower frequencies. For example, in a grayscale histogram, if there are two obvious peaks at grayscale values ​​100 and 150, and a lower valley at 125, this indicates that there are two main grayscale areas in the image. Determine whether there are multiple peaks in the histogram. If so, obtain the valleys between adjacent peaks. The presence of multiple peaks usually means that there are multiple different areas in the image, which may be caused by stains, material differences, etc. For example, if the histogram has a peak at grayscale values ​​of 80 and 180, and a valley at 130, it means that there may be two different grayscale areas in the image, and the valley value 130 is the dividing point between the two areas. The valley value is compared with the preset threshold. If the valley value is lower than the threshold, it is judged that there is a stain on the surface of the residual material. The preset threshold is determined based on experience or experiments and is used to distinguish between normal areas and stain areas. For example, if the preset threshold is 120 and the valley value is 110, it is considered that the area corresponding to the valley value may be a stain. The stain area is marked by the region growing method. The region growing method is a region segmentation algorithm based on seed points. The complete stain area is marked by continuously expanding similar pixels. For example, a pixel with a lower grayscale value is selected as a seed point, and gradually expanded to the surrounding pixels with similar grayscale values, and finally the entire stain area is marked. The marking result is superimposed on the original residual material image to obtain the marked residual material stain image. This step makes the stain area clear in the original image, which is convenient for subsequent analysis and processing. For example, in the marked image, the stain area is highlighted and clearly visible. According to the marked residual material stain image, the characteristic parameters such as the position, size and shape of the stain area are determined. These parameters provide an important basis for subsequent process processing. For example, by measuring the area, perimeter and geometric center position of the stain area, the severity and scope of the stain can be evaluated. The characteristic parameters of the stain area are output as the basis for subsequent process processing. These parameters can help process personnel decide whether to clean or remove the residual material. For example, if the stain area is large and located in a critical position, it may be necessary to remove this part of the residual material; if the stain area is small and does not affect the use, it can be cleaned.

[0080] Obtain the features of the leftover samples such as color, texture, shape and size, and construct a leftover feature vector; randomly select the initial cluster center, and calculate the similarity between each leftover sample and the cluster center; divide the leftover samples into the category of the nearest cluster center according to the similarity; recalculate the cluster center for the leftover samples in each category to obtain a new cluster center; determine whether the new cluster center has changed from the previous cluster center. If the change is less than the preset threshold, the clustering converges and the algorithm ends; otherwise, return to step 3 to continue iterating; analyze the clustering results, extract the typical leftover samples of each category as the representative of the category, and construct a clustered leftover feature database; perform similarity matching between the new leftover samples and the typical samples in the leftover feature database, determine the category to which they belong, and realize rapid classification and management of leftovers.

[0081] The extracted feature parameters are input into a pre-built support vector machine (SVM) model. SVM is a powerful classification algorithm that separates data of different categories by finding the optimal hyperplane. The hypothetical SVM model has been trained with a large number of scrap samples of known categories, and the model contains multiple support vectors and corresponding weights. When the feature parameters of the scrap to be identified are input into the model, the model calculates the distance between these parameters and each support vector based on their position in the feature space, thereby predicting the category to which they belong. The input feature parameters are classified and predicted by the SVM model to obtain the predicted category label of the scrap.

[0082] According to the predicted category label, the corresponding category is retrieved in the pre-clustered residual material feature database. In the assumed residual material feature database, multiple categories have been generated by clustering algorithms (such as K-means), and each category contains the feature data of several typical residual material samples. Through the search, all residual material feature data under the category are found. If a category matching the predicted category label is found in the residual material feature database, the residual material feature data under the category is extracted. Similarity calculation can be performed using a variety of methods, such as Euclidean distance, cosine similarity, etc. Assuming that the Euclidean distance calculation is used, it is found that the distance between the residual material to be identified and a typical sample is the smallest and the similarity is the highest. The residual material type with the highest similarity is determined as the final identification result of the residual material to be identified, and the residual material type is output.

[0083] Through the pre-trained neural network model, the residual material size data of the input layer neurons is used, and the weight calculation of the hidden layer neurons is performed to obtain the inlet size parameters of the collection device corresponding to the output layer.

[0084] A set of pre-collected residual material size data and the corresponding optimal collection device inlet size parameters are obtained as the training data set of the neural network model. A neural network model is constructed, which includes an input layer, at least one hidden layer and an output layer, wherein the number of neurons in the input layer is the same as the dimension of the residual material size data, and the number of neurons in the output layer is the same as the dimension of the collection device inlet size parameters. The neural network model is trained using the training data set, and the connection weights between neurons in each layer are adjusted by the back propagation algorithm, so that the model can fit the nonlinear mapping relationship between the residual material size and the optimal inlet size. The trained neural network model is tested, and a part of the independent residual material size data is used as input to determine whether the collection device inlet size parameters output by the model are close to the actual optimal value. If the error is greater than the preset threshold, the training is continued until the requirements are met. The trained and optimized neural network model is deployed to the production environment. When new residual material size data is input, the model can quickly calculate the corresponding optimal collection device inlet size parameters. According to the output result of the neural network model, the actuator of the collection device is controlled to adjust the inlet size to make it consistent with the calculated optimal parameter value, thereby achieving adaptive optimization. During the production process, data on the size of the residual material and the size of the inlet before and after adjustment are continuously collected, and the neural network model is retrained regularly so that it can adapt to changes in raw material characteristics or process conditions and maintain optimal control performance.

[0085] Obtain information such as the type of residual material, stain area, size parameters, and inlet size parameters as input data for the neural network model. Pass the input data into the pre-trained neural network model, and obtain the predicted result of the motion trajectory of the collection device through forward propagation calculation. Determine the specific motion control instruction sequence of the collection device based on the motion trajectory result predicted by the neural network model. Transmit the generated control instruction sequence to the control unit of the collection device to control the collection device to perform the residual material collection operation according to the specified motion trajectory. During the collection operation of the collection device, the motion state data of the device and the residual material collection situation information are collected in real time through sensors. Compare the collected real-time state data with the predicted motion trajectory, calculate the deviation value, and if the deviation exceeds the preset threshold, trigger the parameter fine-tuning of the neural network model. Use the actual motion data and collection results collected during the collection process to perform incremental training on the neural network model to continuously improve the prediction accuracy and adaptability of the model.

[0086] This embodiment also provides a system for collecting waste materials in cattle and sheep farms based on image recognition, including:

[0087] An image completion module is used to obtain multi-view multi-spectral images of residual materials in a cattle and sheep farm environment, and to perform three-dimensional completion on the residual materials in the multi-view multi-spectral images through three-dimensional reconstruction technology to obtain a completed image;

[0088] An image segmentation module, used for segmenting the completed image based on a region growing segmentation algorithm to obtain a segmented residual image region;

[0089] The residual material classification module is used to extract the characteristic parameters of the residual material image area, classify the residual materials according to the characteristic parameters, and obtain a set of images of different types of residual materials; wherein the characteristic parameters include color histogram, Gabor texture feature and Hu invariant moment;

[0090] A size measurement module, used for extracting the outline of the residual material from the image set to obtain the residual material size information;

[0091] The stain detection module is used to analyze the stain situation on the surface of the residual material and mark the stain area using the region growing method to obtain the marked residual material stain image;

[0092] The clustering update module is used to build a residual material feature database, calculate the similarity between residual material samples, update the cluster center through the similarity matrix, and obtain the residual material feature database after clustering;

[0093] A matching module is used to input the feature parameters of the leftover material to be identified into the support vector machine model to obtain a predicted label, and match the predicted label with the category label in the clustered leftover material feature database to obtain the leftover material type;

[0094] An inlet size adjustment module is used to dynamically adjust the size of the collection device inlet according to the residual material size information;

[0095] The control module is used to input the residual material type, stain area, size information and entrance size into the pre-trained neural network model, generate the motion trajectory control instructions of the collection device through model prediction, and control the collection device to perform the residual material collection operation.

[0096] This embodiment further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0097] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method are implemented.

[0098] This embodiment also provides a computer program product, including a computer program, which implements the steps of the method when executed by a processor.

[0099] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for collecting waste materials in cattle and sheep farms based on image recognition, characterized in that: The following steps are involved: Acquire multi-view multi-spectral images of residual materials in a cattle and sheep farm environment, and use three-dimensional reconstruction technology to perform three-dimensional completion on the residual materials in the multi-view multi-spectral images to obtain a completed image; Segmenting the completed image based on a segmentation algorithm of region growing to obtain a segmented residual material image region; Extract characteristic parameters of the residual material image area, classify the residual materials according to the characteristic parameters, and obtain image sets of different types of residual materials; wherein the characteristic parameters include color histogram, Gabor texture feature and Hu invariant moment; Extracting the outline of the residual material from the image set to obtain residual material size information; Analyze the stains on the surface of the leftover material, and use the region growing method to mark the stain area to obtain the marked leftover material stain image; Construct a waste material feature database, calculate the similarity between waste material samples, update the cluster center through the similarity matrix, and obtain the clustered waste material feature database; Input the feature parameters of the leftovers to be identified into the support vector machine model to obtain a predicted label, and match the predicted label with the category label in the clustered leftover feature database to obtain the leftover type; Dynamically adjust the size of the collection device entrance according to the residual material size information; The type of residual material, stain area, size information and entrance size are input into the pre-trained neural network model, and the motion trajectory control instructions of the collection device are generated through model prediction to control the collection device to perform the residual material collection operation.

2. The method according to claim 1, characterized in that The three-dimensional completion of the residual material in the multi-view multi-spectral image includes: Acquire multi-view and multi-spectral images of leftovers in a cattle and sheep farm environment; pre-process the acquired multi-view and multi-spectral images to remove image noise; use a feature extraction algorithm to extract texture and color features of the leftovers from the images after removing image noise; use a three-dimensional reconstruction algorithm to obtain the three-dimensional spatial structure information of the leftovers based on the extracted leftover features; extract the three-dimensional size features of the leftovers from the three-dimensional spatial structure information of the leftovers; determine whether the leftovers are stacked or blocked by analyzing the three-dimensional spatial position relationship of the leftovers, and if so, use a three-dimensional completion algorithm to perform three-dimensional reconstruction on the determined blocked leftover area to obtain complete three-dimensional structural information of the leftovers.

3. The method according to claim 1, characterized in that Segmenting the completed image includes: By analyzing the color distribution and texture changes of the waste area, the starting point position and growth conditions of the region growing algorithm are determined; starting from the selected starting point, regional growth is performed according to the growth conditions; pixels of similar colors and textures are gradually merged into the same region until no further growth is possible; post-processing is performed on the grown region, including region merging and hole filling, to obtain a complete waste area.

4. The method according to claim 1, characterized in that: The image collections obtained for different types of remnants include: The feature parameters of the residual material image area, including color histogram, Gabor texture features and Hu invariant moments, are extracted. The similarity matrix between the residual material samples is constructed according to the similarity measurement of the feature parameters. The spectral clustering algorithm is used to classify the residual materials. The optimal number of clustering categories is determined by the maximum modularity criterion, and the image collection of different types of residual materials is obtained.

5. The method according to claim 1, characterized in that Obtaining the remaining material size information includes: The Canny edge detection algorithm is used to extract the edge of the image to obtain the edge contour image of the residual material; the polygonal features of the residual material contour are obtained through the contour approximation algorithm; according to the polygonal features, the geometric properties of the polygon are calculated, wherein the geometric properties include the number of sides, side length and internal angle of the polygon; and the size information of the residual material is calculated based on the geometric properties of the polygon.

6. The method according to claim 1, characterized in that Analysis of the surface stains of the residual material includes: Convert the residual material image into a grayscale image; calculate the grayscale histogram of the grayscale image and determine the peak and valley values ​​in the histogram; determine whether there are multiple peaks in the histogram, and if so, obtain the valley value between adjacent peaks; compare the valley value with a preset threshold, and if the valley value is lower than the threshold, determine that there are stains on the surface of the residual material.

7. A system for collecting waste materials in cattle and sheep farms based on image recognition, characterized in that: include: An image completion module is used to obtain multi-view multi-spectral images of residual materials in a cattle and sheep farm environment, and to perform three-dimensional completion on the residual materials in the multi-view multi-spectral images through three-dimensional reconstruction technology to obtain a completed image; An image segmentation module, used for segmenting the completed image based on a region growing segmentation algorithm to obtain a segmented residual image region; The residual material classification module is used to extract the characteristic parameters of the residual material image area, classify the residual materials according to the characteristic parameters, and obtain a set of images of different types of residual materials; wherein the characteristic parameters include color histogram, Gabor texture feature and Hu invariant moment; A size measurement module, used for extracting the outline of the residual material from the image set to obtain the residual material size information; The stain detection module is used to analyze the stain situation on the surface of the residual material and mark the stain area using the region growing method to obtain the marked residual material stain image; The clustering update module is used to build a residual material feature database, calculate the similarity between residual material samples, update the cluster center through the similarity matrix, and obtain the residual material feature database after clustering; A matching module is used to input the feature parameters of the leftover material to be identified into the support vector machine model to obtain a predicted label, and match the predicted label with the category label in the clustered leftover material feature database to obtain the leftover material type; An inlet size adjustment module is used to dynamically adjust the size of the collection device inlet according to the residual material size information; The control module is used to input the residual material type, stain area, size information and entrance size into the pre-trained neural network model, generate the motion trajectory control instructions of the collection device through model prediction, and control the collection device to perform the residual material collection operation.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images