Beef cattle identification method based on multi-scale segmentation optimization and multi-modal data fusion

Abstract

The invention relates to the technical field of intelligent breeding, and discloses a beef cattle identification method based on multi-scale segmentation optimization and multi-modal data fusion, which comprises the following steps: acquiring beef cattle multi-modal image data, carrying out time-space synchronization and fusion to construct a fusion image, carrying out preprocessing of illumination invariant transformation and multi-scale pyramid construction, and carrying out multi-scale segmentation optimization and multi-modal data fusion on the fusion image; generating an enhanced image, inputting the enhanced image into a depth-guided attention segmentation network to extract double-branch foreground features, generating an initial segmentation probability graph, optimizing a segmentation mask in combination with a conditional random field model and a motion consistency constraint, finally extracting multi-dimensional features from the mask, inputting the multi-dimensional features into a multi-classification support vector machine to perform individual classification reasoning, and obtaining a final segmentation result. And obtaining a beef cattle individual identification result. Therefore, the foreground and background segmentation method applied to the intelligent breeding scene for beef cattle individual recognition is provided, the beef cattle individual recognition precision is improved, and meanwhile the management requirements of commercial breeding for high precision, high stability and multi-scene adaptability are met.

Description

Technical Field

[0001] This invention relates to the field of smart farming technology, and in particular to a method for identifying beef cattle based on multi-scale segmentation optimization and multi-modal data fusion. Background Technology

[0002] In modern intelligent beef cattle farming management, the core premise of individual identification is to accurately distinguish between the foreground (beef cattle target) and the background (farming environment, such as grassland, cattle shed facilities, weeds, dynamic light and shadow, etc.).

[0003] However, existing technologies have significant defects in the foreground and background segmentation process, which directly leads to low recognition accuracy in the subsequent process: (1) Traditional fixed-scale segmentation algorithms are prone to losing body details when facing large-scale foreground (near-distance beef cattle), and are prone to being confused with the background when facing small-scale foreground (far-distance beef cattle), resulting in jagged or hollow segmentation boundaries and incomplete foreground feature extraction; (2) In addition, existing solutions rely on RGB images. When the background lighting changes drastically (such as strong light at noon and weak light at night), the foreground texture details in the RGB image are covered by the background light and shadow. When the foreground coat color (such as yellow beef cattle) is similar to the background color (such as withered yellow grass), the grayscale difference between the foreground and the background is reduced. Traditional algorithms such as Gaussian mixture model (GMM) are prone to directly misjudging some foreground as background or misjudging the background as foreground, which seriously affects the subsequent individual feature matching; (3) At the same time, beef cattle recognition is easily affected by the continuity of foreground movement and background dynamic interference, which can easily lead to the body contact area with the ground background being missegmented as background and the foreground contour being broken.

[0004] Therefore, how to provide a foreground and background segmentation method for individual beef cattle identification in smart farming scenarios, which can improve the accuracy of individual beef cattle identification while meeting the management needs of commercial farming for high precision, high stability and multi-scenario adaptability, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This invention provides a method for identifying beef cattle based on multi-scale segmentation optimization and multi-modal data fusion, aiming to solve at least one of the above-mentioned technical problems.

[0006] To achieve the above objectives, this invention provides a method for identifying beef cattle based on multi-scale segmentation optimization and multi-modal data fusion, the method comprising the following steps: S1: Collect multimodal image data of beef cattle in the livestock breeding area, perform spatiotemporal synchronization and data fusion on the multimodal image data, and construct a fused image; S2: Preprocess the fused image to generate an enhanced image; wherein, the preprocessing includes illumination invariant transformation and multi-scale pyramid construction adapted to the dynamic distance of beef cattle; S3: Input the enhanced image into a depth-guided attention segmentation network, extract bi-branch foreground features through a bi-branch encoder, and generate an initial segmentation probability map based on the fused bi-branch foreground features using a multi-scale attention decoder; S4: Based on the conditional random field model and motion consistency constraints, the initial segmentation probability map is physically constrained to optimize the foreground segmentation mask; S5: Extract multi-dimensional features from the foreground segmentation mask, input the multi-dimensional features into a multi-class support vector machine for individual classification reasoning, and output the individual identification result of beef cattle.

[0007] Preferably, step S1: Collecting multimodal image data of beef cattle within the livestock breeding area, performing spatiotemporal synchronization and data fusion on the multimodal image data, and constructing a fused image, specifically including: S11: Utilize RGB cameras, depth cameras, thermal imagers, and LIDAR sensors deployed within the livestock breeding area to collect RGB images, depth images, thermal imaging data, and point cloud data of beef cattle, and use the collected data and collection timestamps to perform spatiotemporal synchronization. S12: The multimodal data, including RGB images, depth images, thermal imaging data, and point cloud data, is calibrated and its coordinate system is unified. A weighted fusion strategy is then used to generate a fused image.

[0008] Preferably, step S12: The multimodal data including RGB images, depth images, thermal imaging data, and point cloud data is calibrated and its coordinate system is unified; a weighted fusion strategy is used to generate a fused image, specifically including: S121: The checkerboard method is used to perform intrinsic parameter calibration on the RGB camera, depth camera and thermal imager respectively, and the checkerboard point cloud data is collected. S122: Perform multimodal matching on checkerboard corner points of RGB images, depth images, thermal imaging data and point cloud data, perform extrinsic parameter calibration of RGB camera, depth camera and thermal imager, and unify depth data, thermal imaging data and point cloud data into RGB image coordinate system; S123: Generate a fused image using a weighted fusion strategy.

[0009] Preferably, in step S2, the illumination invariance transformation specifically includes: S21: Perform global mean-variance normalization on the RGB channel, depth channel, and thermal imaging channel of the fused image to eliminate global brightness differences; S22: The Retinex algorithm is used to perform local illumination compensation calculations on the normalized fused image. The calculation results are then subjected to exponential transformation and normalization to repair the dark foreground areas caused by backlighting and the overexposed foreground areas caused by strong light.

[0010] Preferably, in step S2, the construction of a multi-scale pyramid adapted to the dynamic distance of beef cattle specifically includes: S21: Construct a 4-layer Gaussian pyramid for the image after illumination invariant transformation preprocessing; wherein, constructing a 4-layer Gaussian pyramid includes: taking the image after illumination invariant transformation preprocessing as the 0th layer, performing smoothing processing on each layer image using a 3×3 Gaussian filter, performing downsampling with a step size of 2, and generating the 1st, 2nd and 3rd layers in sequence to obtain a 4-layer Gaussian pyramid. S22: Construct the Laplace pyramid based on the Gaussian pyramid, the specific expression of which is:

[0011] In the formula, Indicates the first Gaussian pyramid Layer image, Indicates to The result obtained after performing upsampling is the same as the first Images with the same resolution at each layer, The first of the Laplace pyramids Layer residual image.

[0012] Preferably, step S3: The enhanced image is input into a depth-guided attention segmentation network, bi-branch foreground features are extracted by a bi-branch encoder, and an initial segmentation probability map is generated based on the fused bi-branch foreground features using a multi-scale attention decoder, specifically including: S31: The enhanced image is input into a depth-guided attention segmentation network, and a dual-branch foreground feature is extracted through a dual-branch encoder; wherein, the dual-branch encoder includes: an RGB branch that takes the Laplacian pyramid layer 0 image as input and the foreground appearance feature as output, and a depth branch that takes the normalized depth image as input and the foreground contour feature as output; S32: Use a cross-attention mechanism to fuse the dual-branch output features to obtain the fused foreground features. The expression for performing fusion on the dual-branch output features is specifically as follows:

[0013] In the formula, This represents the query vector output by the RGB branch. , These represent the key vector and value vector output by the deep branch, respectively. Indicates the dimension of the key vector; S33: Employs a U-Net decoder with an incorporated scale attention module to process the fused foreground features. Perform multi-scale feature weighting calculation, and use the sigmoid activation function to generate an initial segmentation probability map based on the calculation results; the expression for generating the initial segmentation probability map is as follows:

[0014]

[0015] In the formula, Represents pixels The probability of belonging to the foreground. This represents the sigmoid activation function. This represents multi-scale weighted features. Indicates low-level features, Indicates characteristics of high-level personnel.

[0016] Preferably, in step S4, the optimization based on the conditional random field model specifically includes: S41: Construct a fully connected CRF model to perform spatial constraint optimization on the initial segmentation probability map; wherein, the expression for the energy function of the fully connected CRF model is as follows:

[0017] In the formula, Labels representing all pixels, For the future, As background, For the initial segmentation probability map To ensure that the optimization result does not deviate from the univariate potential of the initial segmentation, The binary potential is used to penalize similar pixels with inconsistent labels based on the spatial distance and appearance similarity between pixels i and j. S42: Initialize the pixel label distribution as the initial segmentation probability map Iterate 5 times to update the label distribution using message passing, taking the label with the highest probability in the label distribution of each pixel to generate the foreground segmentation mask after the first optimization; the expression for updating the label distribution using message passing is as follows:

[0018] In the formula, This indicates that in the (t+1)th iteration, pixel i is assigned a label. The probability distribution, Indicates proportional to, This represents the label potential energy of pixel i itself. This indicates that the product operation is performed on all pixels j that are not equal to i. Represents all labels for pixel j Summation, Indicates that the pixel i is labeled as And pixel j is labeled The paired potential energy at time This indicates that in the t-th iteration, pixel j is assigned a label. The probability distribution.

[0019] Preferably, in step S4, the optimization based on motion consistency constraints specifically includes: S43: Optical flow estimation is performed on two consecutive fused images using the FlowNet model, and the flowing optical field is output; wherein, the expression of the flowing optical field is specifically:

[0020] In the formula, Represents the streamer field. , Representing pixels Displacement in the horizontal and vertical directions; S44: The optimized foreground segmentation mask of frame t is mapped to frame t-1 through optical flow field to obtain the predicted foreground region. The overlap rate between the actual foreground region of frame t and the predicted foreground region of frame t-1 is calculated. The expression for the overlap rate is as follows:

[0021] In the formula, Indicates the overlap rate; S45: Perform segmentation result correction based on overlap rate; where the correction logic is: if If the value is greater than the first preset value, the segmentation result of frame t is retained; otherwise, the segmentation probability of pixels in frame t that are predicted to be foreground regions but are actually background is increased by the second preset value, and the segmentation probability of pixels that are predicted to be background regions but are actually foreground regions is decreased by the second preset value, and the foreground label is re-determined.

[0022] Preferably, step S5: extracting multi-dimensional features from the foreground segmentation mask, inputting the multi-dimensional features into a multi-class support vector machine for individual classification inference, and outputting the individual beef cattle identification result, specifically includes: S51: Extract set features, appearance features and spatiotemporal features from the foreground segmentation mask, concatenate the three types of features into multi-dimensional features and perform normalization processing; S52: Based on the multi-dimensional feature vectors of all individual beef cattle generated during the training phase, a beef cattle recognition model based on a multi-class support vector machine is trained after labeling the identity tags. The multi-dimensional features of the beef cattle to be identified are then input into the beef cattle recognition model. S53: Obtain the K-dimensional identity probability vector output by the beef cattle identification model, and take the category with the highest probability as the individual beef cattle identification result; where K represents the total number of beef cattle in the livestock breeding area.

[0023] The beneficial effects of this invention are as follows: It proposes a beef cattle identification method based on multi-scale segmentation optimization and multi-modal data fusion. By collecting multi-modal image data of beef cattle and performing spatiotemporal synchronization and fusion to construct a fused image, and after preprocessing with illumination invariant transformation and multi-scale pyramid construction, an enhanced image is generated and input into a depth-guided attention segmentation network to extract bi-branch foreground features and generate an initial segmentation probability map. Then, a conditional random field model and motion consistency constraints are combined to optimize the segmentation mask. Finally, multi-dimensional features are extracted from the mask and input into a multi-class support vector machine for individual classification inference to obtain the individual beef cattle identification result. Therefore, this invention proposes a foreground and background segmentation method for individual beef cattle identification applied in smart farming scenarios, improving the accuracy of individual beef cattle identification while meeting the management needs of commercial farming for high precision, high stability, and multi-scenario adaptability. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the method for identifying beef cattle based on multi-scale segmentation optimization and multi-modal data fusion according to the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] This invention provides a method for identifying beef cattle based on multi-scale segmentation optimization and multi-modal data fusion, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the method for identifying beef cattle based on multi-scale segmentation optimization and multi-modal data fusion, as described in an embodiment of the present invention.

[0027] In this embodiment, a method for identifying beef cattle based on multi-scale segmentation optimization and multi-modal data fusion is described, the method comprising the following steps: S1: Collect multimodal image data of beef cattle in the livestock breeding area, perform spatiotemporal synchronization and data fusion on the multimodal image data, and construct a fused image; S2: Preprocess the fused image to generate an enhanced image; wherein, the preprocessing includes illumination invariant transformation and multi-scale pyramid construction adapted to the dynamic distance of beef cattle; S3: Input the enhanced image into a depth-guided attention segmentation network, extract bi-branch foreground features through a bi-branch encoder, and generate an initial segmentation probability map based on the fused bi-branch foreground features using a multi-scale attention decoder; S4: Based on the conditional random field model and motion consistency constraints, the initial segmentation probability map is physically constrained to optimize the foreground segmentation mask; S5: Extract multi-dimensional features from the foreground segmentation mask, input the multi-dimensional features into a multi-class support vector machine for individual classification reasoning, and output the individual identification result of beef cattle.

[0028] The existing technology has significant defects in the foreground and background segmentation process, which directly leads to low recognition accuracy in the subsequent process: (1) Traditional fixed-scale segmentation algorithms are prone to losing body details when facing large-scale foreground (near-distance beef cattle) and are prone to confusion with the background when facing small-scale foreground (far-distance beef cattle), resulting in jagged or hollow segmentation boundaries and incomplete foreground feature extraction; (2) In addition, the existing schemes rely on RGB images. When the background lighting changes drastically (such as strong light at noon and weak light at night), the foreground texture details in the RGB image are covered by the background light and shadow. When the foreground coat color (such as yellow beef cattle) is similar to the background color (such as withered yellow grass), the gray difference between the foreground and the background is reduced. Traditional algorithms such as Gaussian mixture model (GMM) are prone to directly misjudging part of the foreground as the background or misjudging the background as the foreground, which seriously affects the subsequent individual feature matching; (3) At the same time, beef cattle recognition is easily affected by the continuity of foreground movement and background dynamic interference, which can easily lead to the body contact area with the ground background being missegmented as the background and the foreground contour being broken.

[0029] In this embodiment, multimodal image data of beef cattle is collected and spatiotemporally synchronized and fused to construct a fused image. After preprocessing with illumination invariant transformation and multi-scale pyramid construction, an enhanced image is generated and input into a depth-guided attention segmentation network to extract dual-branch foreground features and generate an initial segmentation probability map. Then, the segmentation mask is optimized by combining a conditional random field model and motion consistency constraints. Finally, multi-dimensional features are extracted from the mask and input into a multi-class support vector machine for individual classification reasoning to obtain the individual beef cattle identification results.

[0030] In terms of technical principles, this embodiment avoids the problems of losing body details of cattle at close range and confusing cattle with the background at long range by constructing a multi-scale pyramid and extracting features from two branches. It also reduces jagged or hollow segmentation boundaries. By using multimodal data fusion and illumination invariant transformation, it can solve the problems of foreground texture masking caused by drastic changes in illumination and misjudgment when the color of cattle coat is similar to the background color. At the same time, it introduces the spatial constraints of conditional random fields and the temporal constraints of motion consistency to solve the problems of poor continuity of cattle movement, missegmentation of foreground and background regions caused by background dynamic interference, and broken foreground contours. Thus, by proposing a foreground and background segmentation method for individual cattle identification in smart farming scenarios, it can improve the accuracy of individual cattle identification while meeting the management needs of commercial farming for high precision, high stability and multi-scenario adaptability.

[0031] In a feasible embodiment, step S1: Collect multimodal image data of beef cattle within the livestock breeding area, perform spatiotemporal synchronization and data fusion on the multimodal image data, and construct a fused image, specifically including: S11: Utilize RGB cameras, depth cameras, thermal imagers, and LIDAR sensors deployed within the livestock breeding area to collect RGB images, depth images, thermal imaging data, and point cloud data of beef cattle, and use the collected data and collection timestamps to perform spatiotemporal synchronization. S12: The multimodal data, including RGB images, depth images, thermal imaging data, and point cloud data, is calibrated and its coordinate system is unified. A weighted fusion strategy is then used to generate a fused image.

[0032] In this embodiment, RGB images, depth images, thermal imaging data and point cloud data of beef cattle are collected by using RGB cameras, depth cameras, thermal imagers and LIDAR sensors in the livestock breeding area. Spatiotemporal synchronization is performed based on the collection timestamp, and then the multimodal data is calibrated and the coordinate system is unified. A weighted fusion strategy is used to generate a fused image.

[0033] In terms of technical principles, this embodiment first selects an RGB camera (to collect color and texture information), a depth camera (to collect the distance between the target and the sensor and generate a depth image), a thermal imager (to collect the target temperature distribution and generate thermal imaging data), and a LiDAR sensor (to collect 3D point cloud data and record spatial structure) to perform multimodal image data acquisition. Then, by extracting the acquisition timestamp of each type of data, the RGB images, depth images, thermal imaging data, and point cloud data acquired at the same time are correlated to ensure that the multimodal data correspond to the same beef cattle target and scene. Next, a fused image is generated by using data calibration, coordinate system unification, and weighted fusion. While ensuring that the data correspond to the same target, the spatial accuracy of the fused image is improved. It can integrate the appearance details provided by the RGB image, the spatial distance provided by the depth image, the temperature features provided by the thermal imaging data, and the 3D structure provided by the point cloud data to obtain more comprehensive features than a single modality, providing rich information for subsequent preprocessing and segmentation.

[0034] In a feasible embodiment, step S12: calibrating and unifying the coordinate system of the multimodal data including RGB images, depth images, thermal imaging data, and point cloud data, and generating a fused image using a weighted fusion strategy, specifically including: S121: The checkerboard method is used to perform intrinsic parameter calibration on the RGB camera, depth camera and thermal imager respectively, and the checkerboard point cloud data is collected. S122: Perform multimodal matching on checkerboard corner points of RGB images, depth images, thermal imaging data and point cloud data, perform extrinsic parameter calibration of RGB camera, depth camera and thermal imager, and unify depth data, thermal imaging data and point cloud data into RGB image coordinate system; S123: Generate a fused image using a weighted fusion strategy.

[0035] In this embodiment, the weighted fusion strategy is used to generate the fused image, which specifically includes the following execution process: (1) Using a standard checkerboard pattern, place the checkerboard pattern within the sensor's field of view and collect checkerboard pattern images from different angles (at least 10 angles); for RGB cameras, calculate intrinsic parameters (focal length, principal point coordinates, distortion coefficients) by detecting checkerboard corner points; for depth cameras and thermal imagers, perform intrinsic parameter calibration using the same method; and simultaneously collect point cloud data of the checkerboard pattern (for subsequent extrinsic parameter matching). (2) Match the checkerboard corner points (such as the first corner point in the upper left corner and the 80th corner point in the lower right corner) in the RGB image, depth image and thermal imaging data one by one (based on the spatial position and grayscale features of the corner points); calculate the extrinsic parameters (translation vector and rotation matrix) of the depth camera and thermal imager relative to the RGB camera by using the coordinates of the matched corner points as the reference; use the extrinsic parameters to convert the depth data, thermal imaging data and point cloud data to the RGB image coordinate system to ensure that the coordinates of the same spatial point are consistent in the multimodal data.

[0036] (3) Generate a fused image according to a weighted fusion strategy; wherein, the expression of the weighted fusion strategy for generating the fused image is specifically:

[0037] In the formula, This represents the fused image of frame t. This represents the RGB image of frame t. Represents the depth image of frame t. This represents the thermal imaging data of frame t. , , These represent the fusion weights of RGB images, depth images, and thermal imaging data, respectively.

[0038] Therefore, this embodiment uses the checkerboard method to perform intrinsic parameter calibration on the RGB camera, depth camera, and thermal imager and collect checkerboard point cloud data. The checkerboard corner points of the multimodal data are matched to perform extrinsic parameter calibration. Then, based on the weighted fusion formula, the multimodal data are fused to generate a fused image. This integrates the appearance details provided by the RGB image, the spatial distance provided by the depth image, the temperature features provided by the thermal imaging data, and the three-dimensional structure provided by the point cloud data, to obtain more comprehensive features than a single modality, providing rich information for subsequent preprocessing and segmentation.

[0039] In a feasible embodiment, step S2, the illumination invariance transformation, specifically includes: S21: Perform global mean-variance normalization on the RGB channel, depth channel, and thermal imaging channel of the fused image to eliminate global brightness differences; wherein, the expression for performing global mean-variance normalization is as follows:

[0040] In the formula, Indicates the fused image at the pixel level grayscale value at that location This represents the global mean of the fused image. This represents the global standard deviation of the fused image. To prevent the removal of the zero constant.

[0041] S22: The Retinex algorithm is used to perform local illumination compensation calculations on the normalized fused image. The calculation results are then subjected to exponential transformation and normalization to repair the dark foreground areas caused by backlighting and the overexposed foreground areas caused by strong light. The expression for the local illumination compensation calculation is as follows:

[0042] In the formula, This represents the Gaussian convolution kernel. This indicates a convolution operation.

[0043] In this embodiment, global mean-variance normalization (eliminating global brightness differences) is performed on the RGB, depth, and thermal imaging channels of the fused image, respectively. Then, the Retinex algorithm is used for local illumination compensation calculation, and the results are subjected to exponential transformation and normalization to repair backlit dark areas and overexposed areas in strong light. This solves the problem of global imbalance in the fused image caused by brightness differences in different regions (e.g., the left side of the cow's body has a pixel value of 200, while the right side has 100), ensuring consistent brightness for the cattle in different locations. Simultaneously, the Retinex algorithm accurately separates the illumination and reflection components, resolving the problems of dark areas on the cow's head caused by backlighting and overexposed backs caused by strong light, preserving the realistic texture of the cattle.

[0044] In a feasible embodiment, step S2, the construction of a multi-scale pyramid adapted to the dynamic distance of beef cattle, specifically includes: S21: Construct a 4-layer Gaussian pyramid for the image after illumination invariant transformation preprocessing; wherein, constructing a 4-layer Gaussian pyramid includes: taking the image after illumination invariant transformation preprocessing as the 0th layer, performing smoothing processing on each layer image using a 3×3 Gaussian filter, performing downsampling with a step size of 2, and generating the 1st, 2nd and 3rd layers in sequence to obtain a 4-layer Gaussian pyramid. S22: Construct the Laplace pyramid based on the Gaussian pyramid, the specific expression of which is:

[0045] In the formula, Indicates the first Gaussian pyramid Layer image, Indicates to The result obtained after performing upsampling is the same as the first Images with the same resolution at each layer, The first of the Laplace pyramids Layer residual image.

[0046] In this embodiment, a four-layer Gaussian pyramid is constructed from the image after illumination invariance transformation (smoothed by 3×3 Gaussian filtering and downsampled with a step size of 2). Based on the images of each layer of the Gaussian pyramid, the residual image is calculated using the Laplacian pyramid formula to construct a multi-scale pyramid adapted to the dynamic distance of beef cattle. The constructed four-layer Gaussian pyramid covers different scale ranges, solving the problems of losing details at close range and mixing backgrounds at long range in traditional fixed-scale segmentation. The residual image of the Laplacian pyramid is then used to record the details lost by the Gaussian pyramid, improving the subsequent segmentation accuracy.

[0047] In a feasible embodiment, step S3: inputting the enhanced image into a depth-guided attention segmentation network, extracting bi-branch foreground features through a bi-branch encoder, and generating an initial segmentation probability map based on the fused bi-branch foreground features using a multi-scale attention decoder, specifically including: S31: The enhanced image is input into a depth-guided attention segmentation network, and a dual-branch foreground feature is extracted through a dual-branch encoder; wherein, the dual-branch encoder includes: an RGB branch that takes the Laplacian pyramid layer 0 image as input and the foreground appearance feature as output, and a depth branch that takes the normalized depth image as input and the foreground contour feature as output; S32: Use a cross-attention mechanism to fuse the dual-branch output features to obtain the fused foreground features. The expression for performing fusion on the dual-branch output features is specifically as follows:

[0048] In the formula, This represents the query vector output by the RGB branch. , These represent the key vector and value vector output by the deep branch, respectively. Indicates the dimension of the key vector; S33: Employs a U-Net decoder with an incorporated scale attention module to process the fused foreground features. Perform multi-scale feature weighting calculation, and use the sigmoid activation function to generate an initial segmentation probability map based on the calculation results; the expression for generating the initial segmentation probability map is as follows:

[0049]

[0050] In the formula, Represents pixels The probability of belonging to the foreground. This represents the sigmoid activation function. This represents multi-scale weighted features. Indicates low-level features, Indicates characteristics of high-level personnel.

[0051] In this embodiment, the enhanced image is input into a depth-guided attention segmentation network. The dual-branch encoder (RGB branch extracts foreground appearance features, and depth branch extracts foreground contour features) is used to extract dual-branch foreground features. After the features are fused by the cross-attention mechanism, a U-Net decoder with scale attention is used to perform multi-scale weighted calculations. The initial segmentation probability map is generated by combining the sigmoid activation function.

[0052] In practical applications, for feature extraction in dual-branch encoders, a ResNet-50 backbone network is used, divided into two parallel branches. The features of the two branches complement each other (the RGB branch compensates for the lack of appearance discrimination in the depth branch; for example, two cows with similar outlines but different coat colors can be distinguished by the RGB branch; the depth branch compensates for the blurry outlines in low light conditions of the RGB branch; for example, RGB images are blurry at night, but the depth branch can still extract the outlines), thus improving the feature representation capability. Specifically: (1) RGB branch, the input is the Laplacian pyramid layer 0 image (including appearance details), the appearance features of beef cattle (such as coat color and texture) are extracted through convolution and pooling operations, and the output feature map dimension is C×H×W (such as 256×240×480). (2) Depth branch: The input is a normalized depth image (including spatial distance information). Similarly, the contour features of beef cattle (such as trunk edge, head contour, and leg structure) are extracted through ResNet-50. The output feature map dimension is the same as that of the RGB branch (256×240×480).

[0053] For feature fusion using the cross-attention mechanism, cross-attention is used to align dual-branch features, solving the semantic misalignment problem of dual-branch features (such as the misalignment of the cow head feature in the RGB branch and the cow neck feature in the depth branch). Attention weights are used to focus on key regions to improve the accuracy of fused features.

[0054] To generate the segmentation probability map, a U-Net with a scale attention module is used (typically achieved by symmetrical downsampling in the encoder and upsampling in the decoder, fusing high and low-level features through skip connections). The introduction of the scale attention module assigns weights to low-level features (including details, such as ear tags and hair) and high-level features (including semantics, such as the torso) for weighted calculation. The calculated multi-scale weighted features are then subjected to 1×1 convolution (dimensionality reduced to 1×H×W), and an initial segmentation probability map is generated through a sigmoid activation function. The scale attention of the U-Net decoder takes into account both details and semantics, avoiding segmentation noise caused by low-level features (such as background weeds being misidentified as cow hair) and boundary blurring caused by high-level features (such as unclear boundaries between cow legs and the ground).

[0055] In a feasible embodiment, step S4, the optimization based on the conditional random field model, specifically includes: S41: Construct a fully connected CRF model to perform spatial constraint optimization on the initial segmentation probability map; wherein, the expression for the energy function of the fully connected CRF model is as follows:

[0056] In the formula, Labels representing all pixels, For the future, As background, For the initial segmentation probability map To ensure that the optimization result does not deviate from the univariate potential of the initial segmentation, This is a binary potential that penalizes similar pixels with inconsistent labels based on the spatial distance and appearance similarity between pixels i and j.

[0057] In this embodiment, the fully connected CRF model considers the correlation between all pixels, which is different from the traditional sparse CRF. Label optimization is achieved by minimizing the energy function.

[0058] Among them, for the initial segmentation probability map To ensure that the optimization result does not deviate from the univariate potential of the initial segmentation. The formula is: ,Right now The larger, The smaller the value, the lower the energy, and the easier it is to be identified as a foreground element. ).

[0059] Among them, the binary potential for penalizing similar pixels with inconsistent labels based on the spatial distance and appearance similarity between pixels i and j is discussed. The expression is as follows:

[0060] In the formula, This represents the coordinates of pixel i. This represents the grayscale value of pixel i in the fused image. This indicates a tag compatibility function. hour, ,otherwise , , These represent the weights for appearance similarity and spatial distance, respectively. , , These represent the standard deviation of spatial distance, the standard deviation of appearance similarity, and the standard deviation of large spatial distance, respectively.

[0061] S42: Initialize the pixel label distribution as the initial segmentation probability map Iterate 5 times to update the label distribution using message passing, taking the label with the highest probability in the label distribution of each pixel to generate the foreground segmentation mask after the first optimization; the expression for updating the label distribution using message passing is as follows:

[0062] In the formula, This indicates that in the (t+1)th iteration, pixel i is assigned a label. The probability distribution, Indicates proportional to, This represents the label potential energy of pixel i itself. This indicates that the product operation is performed on all pixels j that are not equal to i. Represents all labels for pixel j Summation, Indicates that the pixel i is labeled as And pixel j is labeled The paired potential energy at time This indicates that in the t-th iteration, pixel j is assigned a label. The probability distribution.

[0063] The core logic of updating the label distribution through message passing lies in the fact that the label distribution of each pixel is affected by the label distribution of all other pixels (pixels that are similar in appearance to the cow's body pixels and are close in distance are more likely to be identified as foreground pixels). During the iteration, the rate-limited label distribution is first initialized. After five iterations of updating the label distribution based on message passing, the label with the highest probability is taken for each pixel to generate the first optimized foreground segmentation mask.

[0064] In this embodiment, a fully connected CRF model is constructed. Based on the initial segmentation probability map, an energy function containing univariate and binary potentials is designed to perform spatial constraint optimization. The pixel label distribution is initialized to the initial probability map and then iterated five times with message passing to update the distribution. The maximum probability label of each pixel is taken to generate the first optimized foreground segmentation mask. Thus, by introducing the spatial constraint of the CRF binary potential, the boundary blurring problem of the initial probability map can be solved, and the misjudgment of similar pixels can be corrected. At the same time, the correlation of all pixels can be considered, avoiding local isolation misjudgment caused by traditional segmentation, making the foreground region more complete.

[0065] In a feasible embodiment, step S4, the optimization based on motion consistency constraints, specifically includes: S43: Optical flow estimation is performed on two consecutive fused images using the FlowNet model, and the flowing optical field is output; wherein, the expression of the flowing optical field is specifically:

[0066] In the formula, Represents the streamer field. , Representing pixels Displacement in the horizontal and vertical directions; S44: The optimized foreground segmentation mask of frame t is mapped to frame t-1 through optical flow field to obtain the predicted foreground region. The overlap rate between the actual foreground region of frame t and the predicted foreground region of frame t-1 is calculated. The expression for the overlap rate is as follows:

[0067] In the formula, Indicates the overlap rate; S45: Perform segmentation result correction based on overlap rate; where the correction logic is: if If the value is greater than the first preset value, the segmentation result of frame t is retained; otherwise, the segmentation probability of pixels in frame t that are predicted to be foreground regions but are actually background is increased by the second preset value, and the segmentation probability of pixels that are predicted to be background regions but are actually foreground regions is decreased by the second preset value, and the foreground label is re-determined.

[0068] In this embodiment, the optical flow field is estimated for two consecutive frames of fused images using the FlowNet model. The optimized foreground segmentation mask of frame t is mapped to frame t-1 through optical flow to obtain the predicted foreground region. The overlap ratio (IOU) between the actual foreground region of frame t and the predicted region of frame t-1 is calculated. The segmentation result of frame t is corrected according to the IOU (if the IOU meets the standard, it is retained; otherwise, the pixel segmentation probability is adjusted and the label is re-determined).

[0069] In practical applications, the FlowNet2.0 model is selected for fusing two consecutive frames of images. (Current frame) and (Previous frame) Estimating the stream field To represent Pixels exist Move from China to The position of the foreground is determined to reflect the movement trajectory of the cattle. Then, the foreground segmentation mask of frame t, optimized by CRF, is back-mapped to frame t-1 through optical flow to obtain the predicted foreground region of frame t-1. The overlap rate formula is then used to calculate and measure motion consistency. The range is 0-1, with larger values ​​indicating better motion continuity; after this, a judgment is made. If the value is greater than a first preset value (e.g., 0.7), the segmentation mask of frame t is retained. If it is not greater than the first preset value, the segmentation probability of pixel regions in the predicted foreground region of frame t-1 but which are background in frame t is increased by a second preset value (e.g., 0.2), and the segmentation probability of regions outside the predicted foreground region of frame t-1 but which are foreground in frame t is decreased by a second preset value. The label is then re-determined using sigmoid to generate the corrected mask. This embodiment uses optical flow field to associate continuous frames, avoiding segmentation breaks during cattle movement. When dealing with background dynamic interference, the segmentation probability is calculated based on the mapped data. The segmentation probability is adjusted to avoid misclassification as foreground. At the same time, temporal constraints are used to supplement spatial constraints, and local errors are corrected through motion consistency to ensure the temporal continuity of the segmentation mask and improve the segmentation accuracy of foreground and background.

[0070] In a feasible embodiment, step S5: extracting multi-dimensional features from the foreground segmentation mask, inputting the multi-dimensional features into a multi-class support vector machine for individual classification inference, and outputting the individual beef cattle identification result, specifically includes: S51: Extract set features, appearance features and spatiotemporal features from the foreground segmentation mask, concatenate the three types of features into multi-dimensional features and perform normalization processing; S52: Based on the multi-dimensional feature vectors of all individual beef cattle generated during the training phase, a beef cattle recognition model based on a multi-class support vector machine is trained after labeling the identity tags. The multi-dimensional features of the beef cattle to be identified are then input into the beef cattle recognition model. S53: Obtain the K-dimensional identity probability vector output by the beef cattle identification model, and take the category with the highest probability as the individual beef cattle identification result; where K represents the total number of beef cattle in the livestock breeding area.

[0071] Therefore, in this embodiment, the set features, appearance features and spatiotemporal features are extracted from the foreground segmentation mask and concatenated and normalized. The multi-dimensional features and identity labels of beef cattle in the training phase are used to train a multi-class support vector machine (SVM) model. The features of the beef cattle to be identified are input into the model to obtain a K-dimensional (K is the total number of beef cattle) identity probability vector. The category with the highest probability is taken as the individual beef cattle identification result.

[0072] The features are categorized into three types: ensemble features, which are based on the geometric attributes of the segmentation mask, such as the body area, perimeter, aspect ratio, and centroid coordinates of the cattle; appearance features, which are based on the texture and color attributes of the fused image, such as histogram features of the RGB channels (reflecting coat color distribution) and LBP texture features (reflecting skin wrinkles and hair density); and spatiotemporal features, which are based on the motion attributes of consecutive frames, such as the cattle's speed, direction of movement, and positional changes between adjacent frames. These three types of features are concatenated into a one-dimensional feature vector. During the training phase, a multi-class SVM is used, with the feature vector as input and the identity label as output, to train the cattle recognition model. The normalized multi-dimensional feature vector of the cattle to be identified is then input into the trained SVM model, outputting a K-dimensional identity probability vector (e.g., when K=50, the vector dimension is 50×1, with each element representing the probability that the cattle to be identified belongs to the corresponding number). The category corresponding to the highest probability is taken as the recognition result.

[0073] This invention proposes a method for beef cattle identification based on multi-scale segmentation optimization and multi-modal data fusion. It involves collecting multi-modal image data of beef cattle, performing spatiotemporal synchronization and fusion to construct a fused image, and then preprocessing it through illumination invariant transformation and multi-scale pyramid construction to generate an enhanced image. This enhanced image is then input into a depth-guided attention segmentation network to extract bi-branch foreground features and generate an initial segmentation probability map. The segmentation mask is then optimized using a conditional random field model and motion consistency constraints. Finally, multi-dimensional features are extracted from the mask and input into a multi-class support vector machine for individual classification inference to obtain the individual beef cattle identification result. Therefore, this invention proposes a foreground and background segmentation method for beef cattle identification in smart farming scenarios, improving the accuracy of individual beef cattle identification while meeting the management requirements of high precision, high stability, and multi-scenario adaptability in commercial farming.

[0074] In the description of embodiments of the present invention, specific features, structures, materials or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0075] In the description of the embodiments of the present invention, it should be understood that "-" and "~" represent a range between two numerical values, and this range includes the endpoints. For example, "AB" represents a range greater than or equal to A and less than or equal to B. "A~B" represents a range greater than or equal to A and less than or equal to B.

[0076] In the description of embodiments of the present invention, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0077] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion, characterized in that, The method comprises the following steps: S1: Collecting multi-modal image data of beef cattle in the livestock breeding area, performing space-time synchronization and data fusion on the multi-modal image data, and constructing a fusion image; S2: Preprocessing the fusion image to generate an enhanced image; wherein the preprocessing includes illumination invariance transformation and multi-scale pyramid construction adapting to the dynamic distance of beef cattle; S3: Inputting the enhanced image into a deep guided attention segmentation network, extracting double-branch foreground features through a double-branch encoder, and generating an initial segmentation probability map based on the fused double-branch foreground features using a multi-scale attention decoder; S4: Based on the conditional random field model and the motion consistency constraint, the initial segmentation probability map is physically constrained and optimized to obtain an optimized foreground segmentation mask; S5: Extracting multi-dimensional features from the foreground segmentation mask and inputting the multi-dimensional features into a multi-classification support vector machine for individual classification reasoning to output beef cattle individual recognition results.

2. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 1, wherein, Step S1: Collecting multi-modal image data of beef cattle in the livestock breeding area, performing space-time synchronization and data fusion on the multi-modal image data, and constructing a fusion image, specifically comprising: S11: Using the RGB camera, depth camera, thermal imager and LIDAR sensor deployed in the livestock breeding area, collecting RGB images, depth images, thermal imaging data and point cloud data of beef cattle, and performing space-time synchronization using the collected data and time stamps; S12: Calibrating and unifying the coordinate system of the multi-modal data including RGB images, depth images, thermal imaging data and point cloud data, and generating a fusion image using a weighted fusion strategy.

3. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 2, characterized in that, Step S12: Calibrating and unifying the coordinate system of the multi-modal data including RGB images, depth images, thermal imaging data and point cloud data, and generating a fusion image using a weighted fusion strategy, specifically comprising: S121: Using a checkerboard method to perform intrinsic calibration of the RGB camera, depth camera and thermal imager, and collecting point cloud data of the checkerboard; S122: Multi-modal matching the checkerboard corner points of the RGB image, depth image, thermal imaging data and point cloud data, performing extrinsic calibration of the RGB camera, depth camera and thermal imager, and unifying the depth data, thermal imaging data and point cloud data to the RGB image coordinate system; S123: Generating a fusion image using a weighted fusion strategy.

4. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 1, characterized in that, In step S2, the illumination invariance transformation specifically includes: S21: Performing global mean variance normalization on the RGB channel, depth channel and thermal imaging channel of the fusion image respectively to eliminate global brightness differences; S22: Using the Retinex algorithm to perform local illumination compensation calculation on the normalized fusion image, performing exponential transformation and normalization on the calculation result, and repairing the foreground dark area caused by backlight and the foreground overexposure area caused by strong light.

5. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 1, characterized in that, In step S2, the multi-scale pyramid construction adapting to the dynamic distance of beef cattle specifically includes: S21: constructing a 4-layer Gaussian pyramid for the image preprocessed by the illumination invariance transformation; wherein, the 4-layer Gaussian pyramid is constructed by taking the image preprocessed by the illumination invariance transformation as the 0th layer, performing smoothing processing on each layer of images by using a 3*3 Gaussian filter, performing down-sampling with a step of 2, and sequentially generating the 1st layer, the 2nd layer and the 3rd layer to obtain the 4-layer Gaussian pyramid; S22: constructing a Laplacian pyramid based on the Gaussian pyramid, and the expression is specifically as follows: In the formula, represents the residual image of the first layer of the Laplacian pyramid. represents the image of the first layer of the Gaussian pyramid. represents the image of the first layer of the Gaussian pyramid. represents the image of the first layer of the Gaussian pyramid. represents the image of the first layer of the Gaussian pyramid. represents the residual image of the first layer of the Laplacian pyramid. represents the residual image of the first layer of the Laplacian pyramid.

6. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 5, characterized in that, Step S3: inputting the enhanced image into the deep guided attention segmentation network, extracting double-branch foreground features through a double-branch encoder, and generating an initial segmentation probability map based on the fused double-branch foreground features by using a multi-scale attention decoder, specifically including: S31: inputting the enhanced image into the deep guided attention segmentation network, and extracting double-branch foreground features through a double-branch encoder; wherein, the double-branch encoder includes an RGB branch taking the 0th layer image of the Laplacian pyramid as input and outputting foreground appearance features, and a depth branch taking the normalized depth image as input and outputting foreground contour features; S32: Perform fusion on the double-branch output features by using a cross-attention mechanism to obtain fused foreground features ; wherein the expression for performing fusion on the double-branch output features is specifically: In the formula, denote the query vector output by the RGB branch, 、 denote the key vector and the value vector output by the depth branch, respectively, denote the key vector dimension; S33: using the U-Net decoder with the introduced scale attention module to decode the fused foreground features A multi-scale feature weighting calculation is performed, and a sigmoid activation function is used for the calculation result to generate an initial segmentation probability map; wherein the expression for generating the initial segmentation probability map is specifically: wherein denotes a pixel denotes a probability that a pixel belongs to the foreground denotes a sigmoid activation function denotes a multi-scale weighted feature denotes a low-level feature denotes a high-level feature 7. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 6, characterized in that, In step S4, the optimization based on the conditional random field model includes: S41: constructing a fully connected CRF model to perform spatial constraint optimization on the initial segmentation probability map; wherein, the expression of the energy function of the fully connected CRF model is specifically as follows: wherein, denotes the label of all pixels, is foreground, is background, is based on the initial segmentation probability map ensures that the optimization result does not deviate from the unary potential of the initial segmentation, is a binary potential based on the spatial distance and appearance similarity of pixels i and j, penalizing similar pixels with different labels. S42: initialize the pixel label distribution as an initial segmentation probability map perform message passing update label distribution for 5 iterations, take the label with the maximum probability in each pixel label distribution, and generate a first optimized foreground segmentation mask; wherein the expression of the message passing update label distribution is specific: wherein, P (i | t + 1) denotes the probability distribution of pixel i being assigned a label at iteration t + 1, P (i | t + 1) is proportional to, P (i | t + 1) is proportional to, P (i | t + 1) is proportional to, P (i | t + 1) is proportional to, P (i | t + 1) is proportional to, P (i | t + 1) is proportional to, P (i | t + 1) is proportional to, P (i | t + 1) is proportional to, P (i | t + 1) denotes the probability distribution of pixel i being assigned a label at iteration t + 1.

8. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 7, characterized in that, In step S4, the optimization based on the motion consistency constraint includes: S43: performing optical flow estimation on the fused images of two consecutive frames by using a FlowNet model to output a flow field; wherein, the expression of the flow field is specifically as follows: In the formula, represents a light field, , respectively represent a pixel motion displacement in horizontal, vertical directions; S44: mapping the foreground segmentation mask of the tth frame to the t-1th frame through the flow field to obtain a predicted foreground region, and calculating the overlap rate of the actual foreground region of the tth frame and the predicted foreground region of the t-1th frame; wherein, the expression of the overlap rate is specifically as follows: In the formula, represents the overlap ratio; S45: Perform segmentation result correction based on overlap rate; where the correction logic is: if If the value is greater than the first preset value, the segmentation result of frame t is retained; otherwise, the segmentation probability of pixels in frame t that are predicted to be foreground regions but are actually background is increased by the second preset value, and the segmentation probability of pixels that are predicted to be background regions but are actually foreground regions is decreased by the second preset value, and the foreground label is re-determined.

9. The beef cattle recognition method based on multi-scale segmentation optimization and multi-modal data fusion according to claim 1, characterized in that, Step S5: extracting multi-dimensional features from the foreground segmentation mask, inputting the multi-dimensional features into a multi-classification support vector machine for individual classification reasoning, and outputting a beef cattle individual identification result, specifically including: S51: extracting set features, appearance features and spatio-temporal features from the foreground segmentation mask, concatenating the three types of features into multi-dimensional features and performing normalization processing; S52: taking the multi-dimensional feature vectors of all beef cattle individuals generated in the training phase as samples, labeling the identity tags, training a beef cattle identification model based on the multi-classification support vector machine, and inputting the multi-dimensional features of the beef cattle to be identified into the beef cattle identification model; S53: obtaining a K-dimensional identity probability vector output by the beef cattle identification model, and taking the class with the maximum probability as the beef cattle individual identification result; wherein, K represents the total number of beef cattle in the livestock breeding area.

Images