A comprehensive scoring method for sow body condition based on data layer fusion

Abstract

The present invention discloses a method for comprehensively scoring sow body condition by data layer fusion, which belongs to a detection method and includes the following steps: S1: image acquisition and data acquisition; S2: fusion of RGB image and depth image into point cloud; S3: fusion of original point cloud and fused point cloud; S4: fusion of fused point cloud and one-dimensional data; S5: establishment and training of sow body condition scoring model; S6: sow body condition scoring. The present invention completes the fusion of data layers by one-dimensional data, two-dimensional image and three-dimensional image through a data fusion method, and proposes a CIW method, which aims to improve the model's attention to minority classes (long tail part) while maintaining the recognition ability of majority classes. The RGB image contains the color, contour and texture features of the pig body; the depth camera collects the pig body features from the perspective of a single laser point; the laser radar collects the pig body features from the perspective of multiple distributed laser points; the three different types of image features complement each other, while taking into account different breeds, parities and production stages, so that the scoring method has a wide range of applicability.

Description

Technical Field

[0001] The present invention relates to a detection method, and in particular to a comprehensive scoring method for sow body condition by integrating data layers. Background Art

[0002] Sow body condition is a core indicator for assessing the quality of sow husbandry and production efficiency. Body condition scoring is a quantitative method for assessing animal physical condition, helping to understand animal health, guide husbandry management, and predict animal production performance. Manual visual scoring typically relies on professionals to visually assess the nutritional status and energy reserves of sows by observing the fullness of the buttocks from a posterior view and palpating specific body parts, such as the spine, bones, subcutaneous fat, and muscle. This method is susceptible to subjective factors and therefore has poor reliability. The assessment process is not only time-consuming and labor-intensive, but also prone to causing stress in animals. Backfat thickness scoring, which involves manual measurement of backfat using instruments, also has the disadvantages of being time-consuming, labor-intensive, and prone to causing stress in animals, and is also detrimental to biosafety management. With the recent development of sensor and artificial intelligence technologies, a few studies have explored sow body condition scoring using machine vision. Currently proposed machine learning-based sow body condition scoring methods use only RGB or depth images of the sow's buttocks from a posterior view, resulting in limited accuracy. Currently, there is no method that combines multiple perspectives such as the sow's back, side and buttocks to fuse one-dimensional data, two-dimensional images and three-dimensional images at the data level to obtain comprehensive and integrated data related to the sow's physical condition and then perform sow body condition scoring. Summary of the Invention

[0003] The present invention aims to overcome the deficiencies of the prior art and provides a method for comprehensively scoring sow body condition by integrating data layers.

[0004] A method for comprehensively scoring sow body condition by data layer fusion of the present invention comprises the following steps:

[0005] S1: Image acquisition and data acquisition

[0006] The images and data that need to be acquired during the data acquisition phase include:

[0007] (1) The RGB camera, depth camera, and lidar are used to obtain a two-dimensional RGB image, a three-dimensional depth image, and a three-dimensional point cloud at the same position on the right side, left side, and rear end of the sow, respectively, to achieve image acquisition; wherein the RGB camera and the depth camera are set to the same resolution;

[0008] (2) Obtain one-dimensional data on sow breed, parity, and reproductive stage;

[0009] S2: RGB image and depth image fusion into point cloud

[0010] (1) RGB image registration

[0011] a. Convert the depth map to a grayscale image

[0012] The depth camera is set up at a height of m meters above the ground, so the depth value range in the depth image is [0, m]. The depth image is processed to complete the mapping of depth value to grayscale value, and a grayscale image with depth information is generated. The grayscale value range of the grayscale image is [0, 255], a total of 256 values. The depth value mapping principle is shown in formula (1), where gray pix is the grayscale value of the converted grayscale image, d pix is the depth value in the original depth image, and pix is ​​the subscript of each pixel;

[0013]

[0014] b. Use grayscale image to register RGB image

[0015] The RGB image is matched with the pig body region feature points in the grayscale image obtained in step S2(1)a by using feature point recognition and matching algorithms, and the registration coefficient [c x ,c y ], convert the RGB image so that the pig body in the RGB image and the grayscale image coincide with each other, and obtain the RGB image after registration; the principle of RGB image conversion is shown in formula (2), where (x0, y0) is the coordinate of the RGB image before registration, (x r ,y r ) is the coordinate after registration, c x is the registration coefficient of the horizontal axis, c y is the registration coefficient of the ordinate;

[0016]

[0017] (2) Depth image conversion to point cloud

[0018] Get the intrinsic parameters of the depth camera, focal length f x and f y , convert the data from the image coordinate system to the world coordinate system to obtain the converted point cloud. The calculation formula is shown in formula (3):

[0019]

[0020] Among them, x, y, z are the point cloud coordinates, D is the depth value, and f x and f y is the focal length of the camera, x′ and y′ are the depth image coordinates, and the transformed point cloud contains the background and the pig body;

[0021] (3) Fusion of the registered RGB image and the converted point cloud

[0022] The dimension of the converted point cloud data in S2(2) is n×3, where n is the number of points in each point cloud and 3 is the data dimension of the coordinates (x, y, z) of each point; the dimension of the RGB image after registration in S2(1) is x r ×y r ×3, where x r and y r is the size of the registered RGB image, 3 is the RGB value of each pixel, expressed as (r, g, b);

[0023] The registered RGB image is fused with the converted point cloud through a connection operation, denoted by the symbol ⊕. The fusion operation is defined as (x, y, z) ⊕ (r, g, b), which means that the coordinate point (x, y, z) of the point cloud is connected with the RGB value (r, g, b). The fused data is still presented in the form of a point cloud with a dimension of n × 6, where n is the number of points in the fused point cloud and 6 is the data dimension of each point. The data of each point after fusion is represented as (x, y, z, r, g, b);

[0024] S3: Fusion of original point cloud and fused point cloud

[0025] The fused point cloud obtained by S2 is fused with the original pig point cloud collected by the lidar in step (1) of S1. The two parts of the point cloud contain the pig body and the background respectively. First, the pig point cloud is segmented. The segmented pig body contains the pig's left point cloud, right point cloud and rear hip point cloud respectively. Then all the segmented point clouds are fused. The specific steps are as follows:

[0026] (1) Pig body point cloud segmentation

[0027] The point cloud consists of two parts: the pig body and the background. The KPFCNN method with a flexible convolution kernel is used to segment the pig body point cloud. The background point cloud is removed and the point cloud of the pig body is segmented. The nearest neighbor method is used for point upsampling and the corresponding layer feature propagation is adopted. The core operation of feature extraction is kernel point convolution. The feature of any point o is the sum of the features of all its neighboring points after kernel point convolution. The calculation is shown in formula (4):

[0028]

[0029] Among them, r is the point o in the neighborhood space N o Points inside, f r is the neighborhood point o r Characteristics, g deform is the kernel point convolution operation function. In this model, a flexible convolution kernel is used, and Δ(o) is the offset of the kernel point.

[0030] (2) Point cloud fusion

[0031] The segmented point cloud contains six parts, denoted as i, which are the point clouds of the pig's sides and the pig's buttocks collected by the lidar in step S1 (1), and the point clouds of the pig's sides and the pig's buttocks fused from the RGB and depth images in step S2 (3). The point cloud dimension is n i ×6, i=1,2,3,4,5,6; where n i For the number of points in each point cloud, the six types of point clouds are directly connected, and the dimension of the fused point cloud is

[0032] S4: Fusion of point cloud and one-dimensional data

[0033] The dimension of the point cloud after fusion in step S3 The dimension of one-dimensional data is 1×3. First, the one-dimensional data is repeatedly expanded and the dimension becomes The dimension is The fused point cloud with dimension is The extended data of , the connection operation of the corresponding point features is performed, and the dimension of the final fused data is

[0034] S5: Establishment and training of sow body condition scoring model

[0035] Through the data fusion operations of step S2, step S3, and step S4, the one-dimensional, two-dimensional, and three-dimensional data are fused, and finally the data layer fusion of the multimodal body condition assessment data is completed. The dimension of the fused data is It is used as input to establish a body condition scoring model to complete the sow body condition scoring; the sow body condition scoring adopts a 5-level scoring standard and a body condition scoring method based on Category Influence Weighting (CIW):

[0036] The number of categories here is 5, corresponding to the 5 scoring levels of sows. The specific steps of CIW are as follows:

[0037] (1) Statistics of the number of category samples

[0038] Count the number of samples in each category j and get the number of samples n in the jth category j , j=1,2,3,4,5;

[0039] (2) Forecast uncertainty assessment

[0040] The RepSurf model is used to predict samples of each category. For a single sample, the predicted entropy H is calculated as shown in formula (5):

[0041]

[0042] where p j is the predicted probability of the jth category; in order to calculate the average prediction uncertainty u for each category j j , it is necessary to average the entropy of all samples belonging to this category, and the calculation is shown in formula (6); where the number of samples in the jth category is m j , the entropy of each sample k is H k :

[0043]

[0044] (3) Category influence calculation

[0045] Combined with the number of category samples and prediction uncertainty, calculate the influence I of the jth category j , calculated as shown in formula (7):

[0046]

[0047] (4) Normalized weight

[0048] Normalize the influence I of all categories to get the final weight w of the jth category j , calculated as shown in formula (8):

[0049]

[0050] (5) Application weight

[0051] During model training, the calculated category weights w are used j To adjust the loss function;

[0052] A body condition scoring model (CIW-RepSurf) was constructed based on the CIW method and the basic architecture of RepSurf. It includes a CIW weight calculation module, an input module, a feature extraction module, and an output module. The CIW module completes the weight calculation of different categories, the input module completes the spatial alignment and initialization of the fused data, the feature extraction module extracts features related to the body condition score from the fused data, and the output module completes the prediction of the body condition score. In the output module, the loss value is calculated using the weighted loss of different categories generated by the CIW module.

[0053] S6: Sow body condition scoring

[0054] During the model usage phase, RGB images, depth images, and point clouds of the sow's two sides and rear hips are collected to obtain (input) the sow's breed, parity, and production stage data. The CIW-RepSurf model is used to generate the sow's body condition score, ultimately meeting the sow's body condition score requirements for multiple breeds, parities, and different production stages.

[0055] As a further improvement of the present invention, accurate feeding and management are performed according to the sow body condition score in step S6.

[0056] The present invention provides a comprehensive scoring method for sow body condition based on data layer fusion. The method integrates one-dimensional data, two-dimensional images, and three-dimensional images through the data fusion method proposed in the present invention to complete the data layer fusion, and proposes a CIW method based on category influence weighting. The CIW method dynamically calculates the weight of each category based on the number of category samples and the uncertainty of the model's prediction of the category. This method aims to increase the model's attention to the minority class (the long tail) while maintaining the recognition ability of the majority class. The present invention scores the sow's body condition based on multimodal data from multiple perspectives of the sow. The scoring angles are comprehensive, which is conducive to improving the scoring accuracy. The RGB image contains the color, contour, and texture features of the pig body; the depth camera captures the pig body features from the perspective of a single laser point, mainly the pig body length, width, and height features; the lidar captures the pig body features from the perspective of multiple distributed laser points, mainly the pig body area, volume, angle, etc. The three different types of image features complement each other. At the same time, different breeds, parities, and production stages are taken into account, making the scoring method widely applicable. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The overall process of the data layer fusion sow body condition comprehensive scoring method of the present invention;

[0058] Figure 2 This is the architecture of the CIW-RepSurf body condition scoring model. DETAILED DESCRIPTION

[0059] The present invention provides a method for comprehensively scoring sow body condition by integrating data layers, which is characterized by comprising the following steps:

[0060] S1: Image acquisition and data acquisition

[0061] The images and data that need to be acquired during the data acquisition phase include:

[0062] (1) The RGB camera, depth camera, and lidar are used to obtain a two-dimensional RGB image, a three-dimensional depth image, and a three-dimensional point cloud at the same position on the right side, left side, and rear end of the sow, respectively, to achieve image acquisition; wherein the RGB camera and the depth camera are set to the same resolution;

[0063] (2) Obtain one-dimensional data on sow breed, parity, and reproductive stage;

[0064] S2: RGB image and depth image fusion into point cloud

[0065] (1) RGB image registration

[0066] The resolution of RGB images and depth images is the same, but since the RGB camera is monocular and the depth camera is binocular, although the shooting points are the same, there is an error in the viewing angle, so correction and registration are required.

[0067] a. Convert the depth map to a grayscale image

[0068] The depth camera is installed at a height of m meters from the ground, so the range of depth values ​​in the depth image is [0, m]. By default, the depth image is parsed as a grayscale image, and the depth value is the grayscale value of the grayscale image. The entire image appears close to black. In order to facilitate the selection of feature points from the depth image, the depth image is processed to complete the mapping of depth values ​​to grayscale values, and a grayscale image with depth information is generated. The grayscale value range of the grayscale image is [0, 255], a total of 256 values. The basic idea of ​​mapping depth values ​​to grayscale values ​​is to map the different depth value ranges in the depth image with an interval of [0, m] to the interval [0, 255] to expand the difference between different pixel depths. Only the depth channel value is transformed. The depth value mapping principle is shown in formula (1), where gray pix is the grayscale value of the converted grayscale image, d pix is the depth value in the original depth image, and pix is ​​the subscript of each pixel;

[0069]

[0070] b. Use grayscale image to register RGB image

[0071] The RGB image is matched with the pig body region feature points in the grayscale image obtained in step S2(1)a by using feature point recognition and matching algorithms, and the registration coefficient [c x ,c y ], convert the RGB image so that the pig body in the RGB image and the depth image coincide with each other, and obtain the RGB image after registration; the principle of RGB image conversion is shown in formula (2), where (x0, y0) is the coordinate of the RGB image before registration, (x r ,y r ) is the coordinate after registration, c x is the registration coefficient of the horizontal axis, c y is the registration coefficient of the ordinate;

[0072]

[0073] (2) Depth image conversion to point cloud

[0074] Get the intrinsic parameters of the depth camera, focal length f x and f y , according to the camera parameter focal length, the data is converted from the image coordinate system to the world coordinate system to obtain the converted point cloud. The calculation formula is shown in formula (3):

[0075]

[0076] Among them, x, y, z are the point cloud coordinates, D is the depth value, and f x and f y is the focal length of the camera, x′ and y′ are the depth image coordinates, and the transformed point cloud contains the background and the pig body;

[0077] (3) Fusion of the registered RGB image and the converted point cloud

[0078] The dimension of the converted point cloud data in S2(2) is n×3, where n is the number of points in each point cloud and 3 is the data dimension of the coordinates (x, y, z) of each point; the dimension of the RGB image after registration in S2(1) is x r ×y r ×3, where x r and y r is the size, i.e., resolution, of the registered RGB image, and 3 is the RGB value of each pixel, expressed as (r, g, b). Since the point cloud is converted from the depth image and the RGB image is registered from the depth image, the two images have the same coordinates x and y.

[0079] The registered RGB image is fused with the converted point cloud through a connection operation, denoted by the symbol ⊕. The fusion operation is defined as (x, y, z) ⊕ (r, g, b), which means that the coordinate point (x, y, z) of the point cloud is connected with the RGB value (r, g, b). The fused data is still presented in the form of a point cloud with a dimension of n × 6, where n is the number of points in the fused point cloud and 6 is the data dimension of each point. The data of each point after fusion is represented as (x, y, z, r, g, b);

[0080] S3: Fusion of original point cloud and fused point cloud

[0081] The fused point cloud obtained by S2 is fused with the original pig point cloud collected by the lidar in step (1) of S1. The two parts of the point cloud contain the pig body and the background respectively. First, the pig point cloud is segmented. The segmented pig body contains the pig's left point cloud, right point cloud and rear hip point cloud respectively. Then all the segmented point clouds are fused. The specific steps are as follows:

[0082] (1) Pig body point cloud segmentation

[0083] The point cloud consists of two parts: the pig body and the background. The KPFCNN method with a flexible convolution kernel is used to segment the pig body point cloud. The background point cloud is removed and the point cloud of the pig body is segmented. The nearest neighbor method is used for point upsampling and the corresponding layer feature propagation is adopted. The core operation of feature extraction is kernel point convolution. The feature of any point o is the sum of the features of all its neighboring points after kernel point convolution. The calculation is shown in formula (4):

[0084]

[0085] Among them, r is the point o in the neighborhood space N o Points inside, f r is the neighborhood point o r Characteristics, g deform is the kernel point convolution operation function. A flexible convolution kernel is used in this model, and Δ(o) is the offset of the kernel point. The model can generate a corresponding offset vector to change the spatial position of the kernel point according to the local structure of the point cloud, thereby generating a flexible convolution kernel that adapts to the local point cloud structure. Ultimately, the model can extract features from pig point clouds of different shapes.

[0086] (2) Point cloud fusion

[0087] The segmented point cloud contains six parts, denoted as i, which are the point clouds of the pig's sides and the pig's buttocks collected by the lidar in step S1 (1), and the point clouds of the pig's sides and the pig's buttocks fused from the RGB and depth images in step S2 (3). The point cloud dimension is n i ×6, i=1,2,3,4,5,6; where n i For the number of points in each point cloud, the six types of point clouds are directly connected, and the dimension of the fused point cloud is

[0088] S4: Fusion of point cloud and one-dimensional data

[0089] The dimension of the point cloud after fusion in step S3 The dimension of one-dimensional data is 1×3. First, the one-dimensional data is repeatedly expanded and the dimension becomes The dimension is The fused point cloud with dimension is The extended data of , the connection operation of the corresponding point features is performed, and the dimension of the final fused data is

[0090] S5: Establishment and training of sow body condition scoring model

[0091] Through the data fusion operations of step S2, step S3, and step S4, the one-dimensional, two-dimensional, and three-dimensional data are fused, and finally the data layer fusion of the multimodal body condition assessment data is completed. The dimension of the fused data is This is used as input to establish a body condition scoring model to complete the sow body condition scoring; the sow body condition scoring adopts a 5-level scoring standard. Since sows are generally fed with precision feeding and limited feeding, the body condition scores of sows in pig farms are concentrated between 1 and 2. As the score increases, the number of samples decreases significantly, and the number of sows with a score of 5 is already very small, showing a typical long-tail distribution characteristic. The sow body condition scoring adopts a 5-level scoring standard. In this application, to address this problem, a body condition scoring method based on Category Influence Weighting (CIW) is proposed:

[0092] The number of categories here is 5, corresponding to the 5 scoring levels of sows. The specific steps of CIW are as follows:

[0093] (1) Statistics of the number of category samples

[0094] Count the number of samples in each category j and get the number of samples n in the jth category j , j=1,2,3,4,5;

[0095] (2) Forecast uncertainty assessment

[0096] The RepSurf model is used to predict samples of each category. For a single sample, the predicted entropy H is calculated as shown in formula (5).

[0097]

[0098] where p j is the predicted probability of the jth category; in order to calculate the average prediction uncertainty u for each category j j , it is necessary to average the entropy of all samples belonging to this category, and the calculation is shown in formula (6); where the number of samples in the jth category is m j , the entropy of each sample k is H k :

[0099]

[0100] (3) Category influence calculation

[0101] Combined with the number of category samples and prediction uncertainty, calculate the influence I of the jth category j , calculated as shown in formula (7):

[0102]

[0103] (4) Normalized weight

[0104] Normalize the influence I of all categories to get the final weight w of the jth category j , calculated as shown in formula (8):

[0105]

[0106] (5) Application weight

[0107] During model training, the calculated category weights w are used j To adjust the loss function so that the model pays more attention to minority classes and classes with high prediction uncertainty;

[0108] A body condition scoring model (CIW-RepSurf) was constructed based on the CIW method and the RepSurf basic framework. The overall structure of the model is as follows: Figure 2 As shown in Figure 1, it includes CIW weight calculation module, input module, feature extraction module and output module. First, the dimension of the data finally fused by S5 is The three-dimensional data is used as the input data of the model, and the spatial alignment and initialization of the fused data are completed through the input module. Then, the data initialized by the input module is passed to the feature extraction module, and the feature extraction module extracts features related to the physical condition score from the fused data. At the same time, the weight calculation of different categories is completed through the CIW module. Specifically, the number of five classification samples with scores of 1-5 in the physical condition score samples is counted separately, and the weights w1, w2, w3, w4, and w5 of the five categories are calculated using the CIW method. Finally, the output features of the feature extraction module and the five weight values ​​of the CIW module are passed to the output module. The output module further aggregates the received features. At the same time, in the output module, the loss value is calculated using the weighted loss of different categories generated by the CIW module. The calculation of the loss value is shown in formula (9). Among them, L CIW (p,y) is the final loss function of the CIW-RepSurf model, L is the loss function before the class loss weighting, y is the five scoring categories, y∈{1,2,3,4,5}. p is the probability predicted by the model for different categories, then p=[p1,p2,p3,p4,p5] T , and p∈[0,1]. w is the loss weight calculated using CIW, w=[w1,w2,w3,w4,w5], w∈[0,1].

[0109] L CIW (p,y)=L(w×p,y) (9)S6:Sow body condition score

[0110] During the model usage phase, RGB images, depth images, and point clouds of the sow's two sides and rear hips are collected to obtain (input) the sow's breed, parity, and production stage data. The CIW-RepSurf model is used to generate the sow's body condition score, ultimately meeting the body condition scores of sows of multiple breeds, parities, and different production stages. Accurate feeding and management are carried out based on the individual sow's body condition score, maximizing the sow's production capacity and achieving maximum production benefits.

Claims

1. A method for comprehensive scoring of sow body condition based on data layer fusion, which includes the following features: The steps are as follows: S1: Image acquisition and data acquisition The images and data that need to be acquired during the data acquisition phase include: (1) The RGB camera, depth camera, and lidar are used to obtain a two-dimensional RGB image, a three-dimensional depth image, and a three-dimensional point cloud at the same position on the right side, left side, and rear end of the sow, respectively, to achieve image acquisition; wherein the RGB camera and the depth camera are set to the same resolution; (2) Obtain one-dimensional data on sow breed, parity, and reproductive stage; S2: RGB image and depth image fusion into point cloud (1) RGB image registration a. Convert the depth map to a grayscale image The depth camera is set up at a height of m meters above the ground, so the depth value range in the depth image is [0, m]. The depth image is processed to complete the mapping of depth value to grayscale value, and a grayscale image with depth information is generated. The grayscale value range of the grayscale image is [0, 255], a total of 256 values. The depth value mapping principle is shown in formula (1), where gray pix is the grayscale value of the converted grayscale image, d pix is the depth value in the original depth image, and pix is ​​the subscript of each pixel; b. Use grayscale image to register RGB image The RGB image is matched with the pig body region feature points in the grayscale image obtained in step S2(1)a by using feature point recognition and matching algorithms, and the registration coefficient [c x ,c y ], convert the RGB image so that the pig body in the RGB image and the depth image coincide with each other, and obtain the RGB image after registration; the principle of RGB image conversion is shown in formula (2), where (x0, y0) is the coordinate of the RGB image before registration, (x r ,y r ) is the coordinate after registration, c x is the registration coefficient of the horizontal axis, c y is the registration coefficient of the ordinate; (2) Depth image conversion to point cloud Get the intrinsic parameters of the depth camera, focal length f x and f y , convert the data from the image coordinate system to the world coordinate system to obtain the converted point cloud. The calculation formula is shown in formula (3): Among them, x, y, z are the point cloud coordinates, D is the depth value, and f x and f y is the focal length of the camera, x′ and y′ are the depth image coordinates, and the transformed point cloud contains the background and the pig body; (3) Fusion of the registered RGB image and the converted point cloud The dimension of the converted point cloud data in S2(2) is n×3, where n is the number of points in each point cloud and 3 is the data dimension of the coordinates (x, y, z) of each point; the dimension of the RGB image after registration in S2(1) is x r ×y r ×3, where x r and y r is the size of the registered RGB image, 3 is the RGB value of each pixel, expressed as (r, g, b); The registered RGB image is fused with the converted point cloud through a connection operation, denoted by the symbol ⊕. The fusion operation is defined as (x, y, z) ⊕ (r, g, b), which means that the coordinate point (x, y, z) of the point cloud is connected with the RGB value (r, g, b). The fused data is still presented in the form of a point cloud with a dimension of n × 6, where n is the number of points in the fused point cloud and 6 is the data dimension of each point. The data of each point after fusion is represented as (x, y, z, r, g, b); S3: Fusion of original point cloud and fused point cloud The fused point cloud obtained by S2 is fused with the original pig point cloud collected by the lidar in step (1) of S1. The two parts of the point cloud contain the pig body and the background respectively. First, the pig point cloud is segmented. The segmented pig body contains the pig's left point cloud, right point cloud and rear hip point cloud respectively. Then all the segmented point clouds are fused. The specific steps are as follows: (1) Pig body point cloud segmentation The point cloud consists of two parts: the pig body and the background. The KPFCNN method with a flexible convolution kernel is used to segment the pig body point cloud. The background point cloud is removed and the point cloud of the pig body is segmented. The nearest neighbor method is used for point upsampling and the corresponding layer feature propagation is adopted. The core operation of feature extraction is kernel point convolution. The feature of any point o is the sum of the features of all its neighboring points after kernel point convolution. The calculation is shown in formula (4): Among them, r is the point o in the neighborhood space N o Points inside, f r is the neighborhood point o r Characteristics, g deform is the kernel point convolution operation function. A flexible convolution kernel is used in this model, and Δ(o) is the offset of the kernel point. The model can generate a corresponding offset vector to change the spatial position of the kernel point according to the local structure of the point cloud, thereby generating a flexible convolution kernel that adapts to the local point cloud structure. Ultimately, the model can extract features from pig point clouds with different shapes. (2) Point cloud fusion The segmented point cloud contains six parts, denoted as i, which are the point clouds of the pig's sides and the pig's buttocks collected by the lidar in step S1 (1), and the point clouds of the pig's sides and the pig's buttocks fused from the RGB and depth images in step S2 (3). The point cloud dimension is n i ×6, i=1,2,3,4,5,6; where n i For the number of points in each point cloud, the six types of point clouds are directly connected, and the dimension of the fused point cloud is S4: Fusion of point cloud and one-dimensional data The dimension of the point cloud after fusion in step S3 The dimension of one-dimensional data is 1×3. First, the one-dimensional data is repeatedly expanded and the dimension becomes The dimension is The fused point cloud with dimension is The extended data of , the connection operation of the corresponding point features is performed, and the dimension of the final fused data is S5: Establishment and training of sow body condition scoring model Through the data fusion operations of step S2, step S3, and step S4, the one-dimensional, two-dimensional, and three-dimensional data are fused, and finally the data layer fusion of the multimodal body condition assessment data is completed. The dimension of the fused data is It is used as input to establish a body condition scoring model to complete the sow body condition scoring; the sow body condition scoring adopts a 5-level scoring standard and a body condition scoring method based on the weighted category influence: The number of categories here is 5, corresponding to the 5 scoring levels of sows. The specific steps of CIW are as follows: (1) Statistics of sample size of each category Count the number of samples in each category j and get the number of samples n in the jth category j , j=1,2,3,4,5; (2) Forecast uncertainty assessment The RepSurf model is used to predict samples of each category. For a single sample, the predicted entropy H is calculated as shown in formula (5): where p j is the predicted probability of the jth category; in order to calculate the average prediction uncertainty u for each category j j , it is necessary to average the entropy of all samples belonging to this category, and the calculation is shown in formula (6); where the number of samples in the jth category is m j , the entropy of each sample k is H k : (3) Category influence calculation Combined with the number of category samples and prediction uncertainty, calculate the influence I of the jth category j , calculated as shown in formula (7): (4) Normalized weight Normalize the influence I of all categories to get the final weight w of the jth category j , calculated as shown in formula (8): (5) Application weight During model training, the calculated category weight w is used j To adjust the loss function; Based on the CIW method and the basic architecture of RepSurf, a body condition scoring model is constructed, including a CIW weight calculation module, an input module, a feature extraction module, and an output module. The CIW module completes the weight calculation of different categories, the input module completes the spatial alignment and initialization of the fused data, the feature extraction module extracts features related to the body condition score from the fused data, and the output module completes the prediction of the body condition score. In the output module, the loss value is calculated using the weighted loss of different categories generated by the CIW module. S6: Sow body condition score During the model usage phase, RGB images, depth images, and point clouds of the sow's two sides and rear hips are collected to obtain the sow's breed, parity, and production stage data. The CIW-RepSurf model is used to generate the sow's body condition score, ultimately meeting the body condition score requirements for sows of multiple breeds, parities, and different production stages.

2. A method for comprehensively scoring sow body condition by data layer fusion according to claim 1, characterized in that According to the sow body condition score in step S6, accurate feeding and management are carried out.

Images