Sow body condition comprehensive scoring method based on data layer fusion

Abstract

The invention discloses a sow body condition comprehensive scoring method based on data layer fusion, and belongs to a detection method. The method comprises the following steps: S1, image acquisition and data acquisition; s2, fusing the RGB image and the depth image into a point cloud; s3, fusing the original point cloud and the fused point cloud; s4, fusing the fused point cloud and the one-dimensional data; s5, establishing and training a sow body condition scoring model; s6, scoring the body conditions of the sows. According to the method, data layer fusion of one-dimensional data, two-dimensional images and three-dimensional images is completed through a data fusion method, a CIW method is provided, the method aims at improving the attention degree of a model on minority classes (long tail parts) and maintaining the recognition capacity on majority classes, and RGB images comprise the color, contour and texture features of pig bodies; pig body characteristics of a single laser point visual angle are collected by the depth camera; the laser radar collects pig body characteristics of the visual angles of a plurality of distributed laser points; the three types of different image features complement each other, and meanwhile, different varieties, tire numbers and production stages are considered, so that the scoring method has wide applicability.

Description

Technical Field

[0001] The invention relates to a detection method, and in particular to a sow body condition comprehensive scoring method with data layer fusion. Background Art

[0002] The body condition of sows is a core indicator for evaluating the quality of their feeding and management and production efficiency. Body condition scoring is a quantitative method for evaluating the physical condition of animals, which helps to understand the health status of animals, guide feeding and management, and predict the production performance of animals. The manual visual scoring method usually relies on professionals to determine the nutritional status and energy reserves of sows by observing the fullness of the buttocks in the rear view of the sow and touching specific parts of the body such as the spine of the sow, such as bones, subcutaneous fat and muscles, and to make intuitive scores. The results of this scoring method are easily affected by subjective factors, so the reliability is poor. The assessment process is not only time-consuming and labor-intensive, but also easy to cause stress reactions in animals. The manual measurement of backfat using instruments through backfat thickness scoring also has the disadvantages of being time-consuming and labor-intensive and easy to cause stress reactions in animals, and is not conducive to biosafety management. In recent years, with the development of sensor technology and artificial intelligence technology, a few studies have explored sow body condition scoring through machine vision technology. The sow body condition scoring method based on machine learning currently proposed only uses RGB images or depth images of the rear view of the sow buttocks to score body condition, and the accuracy of the scoring is limited. Currently, there is no method that combines multiple perspectives such as the sow's back, side and buttocks to fuse one-dimensional data, two-dimensional images and three-dimensional images at the data level to obtain comprehensive and integrated data related to the sow's physical condition, and then perform body condition scoring on the sow. Summary of the invention

[0003] The present invention aims to overcome the deficiencies of the prior art and provides a method for comprehensively scoring sow body condition by fusion of data layers.

[0004] A method for comprehensively scoring sow body condition by data layer fusion of the present invention comprises the following steps:

[0005] S1: Image acquisition and data acquisition

[0006] The images and data that need to be acquired during the data acquisition phase include:

[0007] (1) A 2D RGB image, a 3D depth image, and a 3D point cloud of the same position of the sow are acquired by using an RGB camera, a depth camera, and a laser radar respectively on the right side, the left side, and the back of the buttocks to realize image acquisition; wherein the RGB camera and the depth camera are set to the same resolution;

[0008] (2) Obtain one-dimensional data on sow breed, parity, and reproductive stage;

[0009] S2: RGB image and depth image are fused into point cloud

[0010] (1) RGB image registration

[0011] a. Convert the depth map to grayscale

[0012] The depth camera is installed at a height of m meters above the ground, so the depth value range in the depth image is [0, m]. The depth image is processed to complete the mapping of depth value to grayscale value, and a grayscale image with depth information is generated. The grayscale value range of the grayscale image is [0, 255], a total of 256 values. The depth value mapping principle is shown in formula (1), where z i is the grayscale value of the converted grayscale image, d i is the depth value in the original depth image, and i is the subscript of each pixel;

[0013]

[0014] b. Use grayscale image to register RGB image

[0015] The RGB image is matched with the pig region feature points in the grayscale image obtained in step S2(1)a by using feature point recognition and matching algorithms, and the registration coefficient [c x ,c y ], convert the RGB image so that the pig body in the RGB image and the grayscale image overlap, and obtain the registered RGB image; the principle of RGB image conversion is shown in formula (2), where (x0, y0) is the coordinate of the RGB image before registration, (x, y) is the coordinate after registration, c x is the registration coefficient of the horizontal axis, c y is the registration coefficient of the ordinate;

[0016]

[0017] (2) Depth image conversion to point cloud

[0018] Get the internal parameters of the depth camera, focal length f x and f y , transform the data from the image coordinate system to the world coordinate system to obtain the transformed point cloud. The calculation formula is shown in formula (3):

[0019]

[0020] Among them, x, y, z are the point cloud coordinates, D is the depth value, and f x and f y is the focal length of the camera, x′ and y′ are the depth image coordinates, and the transformed point cloud contains the background and the pig body;

[0021] (3) Fusion of the registered RGB image and the converted point cloud

[0022] The data dimension of the converted point cloud in S2(2) is n×3, where n is the number of points in each point cloud and 3 is the data dimension of the coordinates (x, y, z) of each point. The data dimension of the aligned RGB image in S2(1) is x×y×3, where x and y are the sizes of the aligned RGB image and 3 is the RGB value of each pixel, expressed as (r, g, b).

[0023] The registered RGB image is fused with the converted point cloud through the connection operation, which is denoted by the symbol ⊕. The fusion operation is defined as (x, y, z) ⊕ (r, g, b), which means connecting the coordinate point (x, y, z) of the point cloud with the RGB value (r, g, b). The fused data is still presented in the form of a point cloud. The dimension of the point cloud is n×6, where n is the number of points in the fused point cloud and 6 is the data dimension of each point. The data of each point after fusion is represented as (x, y, z, r, g, b).

[0024] S3: Fusion of original point cloud and fused point cloud

[0025] The fused point cloud obtained by S2 is fused with the original pig point cloud collected by the laser radar in step (1) of S1. The two parts of the point cloud include the pig body and the background respectively. First, the pig point cloud is segmented. The segmented pig body includes the left point cloud, the right point cloud and the rear hip point cloud of the pig respectively. Then, all the segmented point clouds are fused. The specific steps are as follows:

[0026] (1) Pig body point cloud segmentation

[0027] The point cloud consists of two parts: the pig body and the background. The KPFCNN method with a flexible convolution kernel is used to segment the pig body point cloud. The background point cloud is removed and the point cloud of the pig body is segmented. The nearest neighbor method is used for point upsampling and the corresponding layer feature propagation is used. The core operation of feature extraction is kernel point convolution. The feature of any point x is the sum of the features of all its neighboring points after kernel point convolution. The calculation is shown in formula (4):

[0028]

[0029] Among them, x i is the point x in the neighborhood space N x The point inside, f i is the neighborhood point x i The function g is the kernel point convolution operation. In this model, a flexible convolution kernel is used, and Δ(x) is the offset of the kernel point.

[0030] (2) Point cloud fusion

[0031] The segmented point cloud contains six parts, denoted as i, which are the point clouds of the pig's sides and the pig's rear hip collected by the laser radar in step S1 (1), and the point clouds of the pig's sides and the pig's rear hip fused by the RGB and depth images in step S2 (3). The point cloud dimension is n i ×6, i=1,2,3,4,5,6; where n i For the number of points in each point cloud, the six types of point clouds are directly connected, and the dimension of the fused point cloud is

[0032] S4: Fusion of point cloud and one-dimensional data

[0033] The dimension of the point cloud after fusion in step S3 The dimension of one-dimensional data is 1×3. First, the one-dimensional data is repeatedly expanded and the dimension becomes The dimension is The fused point cloud with dimension is The extended data of is used to connect the corresponding point features. The dimension of the final fused data is

[0034] S5: Establishment and training of sow body condition scoring model

[0035] Through the data fusion operations in steps S2, S3, and S4, the one-dimensional, two-dimensional, and three-dimensional data are fused, and finally the data layer fusion of the multimodal body condition assessment data is completed. The dimensions of the fused data are It is used as input to establish a body condition scoring model to complete the sow body condition scoring; the sow body condition scoring adopts a 5-level scoring standard and a body condition scoring method based on Category Influence Weighting (CIW):

[0036] The number of categories here is 5, corresponding to the 5 scoring levels of sows. The specific steps of CIW are as follows:

[0037] (1) Statistics of sample quantity of each category

[0038] Count the number of samples in each category i and get the number of samples n in the i-th category i , i=1,2,3,4,5;

[0039] (2) Forecast uncertainty assessment

[0040] The RepSurf model is used to predict samples of each category. For a single sample, the predicted entropy H is calculated as follows:

[0041] As shown in formula 5:

[0042]

[0043] where p i is the predicted probability of the i-th category; in order to calculate the average prediction uncertainty u for each category i i , it is necessary to average the entropy of all samples belonging to this category, and the calculation is shown in Formula 6; where the number of samples in the i-th category n i , the entropy of each sample k is H k :

[0044]

[0045] (3) Category influence calculation

[0046] Combine the number of category samples and prediction uncertainty to calculate the influence I of the i-th category i , calculated as shown in formula (7):

[0047]

[0048] (4) Normalized weight

[0049] Normalize the influence I of all categories to get the final weight w of the i-th category i , calculated as shown in formula (8):

[0050]

[0051] (5) Application weight

[0052] During model training, the calculated category weights w are used i To adjust the loss function;

[0053] A body condition score model (CIW-RepSurf) is constructed based on the CIW method and the basic architecture of RepSurf, including a CIW weight calculation module, an input module, a feature extraction module, and an output module. The CIW module completes the weight calculation of different categories, the input module completes the spatial alignment and initialization of the fused data, the feature extraction module extracts features related to the body condition score from the fused data, and the output module completes the prediction of the body condition score. In the output module, the loss value is calculated using the weighted losses of different categories generated by the CIW module.

[0054] S6: Sow body condition scoring

[0055] During the model usage phase, RGB images, depth images and point clouds of the sow’s two sides and rear hips are collected to obtain (input) the sow’s breed, parity and production stage data, and the CIW-RepSurf model is used to generate the sow’s body condition score, ultimately meeting the requirements for the body condition score of sows of multiple breeds and parities at different production stages.

[0056] As a further improvement of the present invention, accurate feeding and management are performed according to the sow body condition score in step S6.

[0057] The present invention discloses a method for comprehensively scoring the body condition of sows by data layer fusion. The method of data fusion proposed in the present invention integrates one-dimensional data, two-dimensional images and three-dimensional images to complete the data layer fusion, and proposes a CIW method based on weighted category influence. The CIW method dynamically calculates the weight of each category based on the number of category samples and the uncertainty of the model's prediction of the category. This method aims to improve the model's attention to the minority category (long tail part) while maintaining the recognition ability of the majority category. The present invention scores the body condition of sows based on multimodal data from multiple perspectives of sows, and the scoring angle is comprehensive, which is conducive to improving the scoring accuracy. The RGB image contains the color, contour and texture features of the pig body; the depth camera collects the pig body features from the perspective of a single laser point, mainly the pig body length, width and height features; the laser radar collects the pig body features from the perspective of multiple distributed laser points, mainly the pig body area, volume, angle and other features; the three different types of image features complement each other. At the same time, different breeds, parities and production stages are taken into account, so that the scoring method has a wide range of applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The overall process of the method for comprehensive scoring of sow body condition by data layer fusion of the present invention;

[0059] Figure 2 The architecture of the CIW-RepSurf body condition scoring model. DETAILED DESCRIPTION

[0060] A method for comprehensively scoring sow body condition by data layer fusion of the present invention is characterized by comprising the following steps:

[0061] S1: Image acquisition and data acquisition

[0062] The images and data that need to be acquired during the data acquisition phase include:

[0063] (1) A 2D RGB image, a 3D depth image, and a 3D point cloud of the same position of the sow are acquired by using an RGB camera, a depth camera, and a laser radar respectively on the right side, the left side, and the back of the buttocks to realize image acquisition; wherein the RGB camera and the depth camera are set to the same resolution;

[0064] (2) Obtain one-dimensional data on sow breed, parity, and reproductive stage;

[0065] S2: RGB image and depth image are fused into point cloud

[0066] (1) RGB image registration

[0067] The resolution of RGB image and depth image is the same, but because the RGB camera is monocular and the depth camera is binocular, although the shooting point is the same, there is an error in the viewing angle, so correction and registration are required.

[0068] a. Convert the depth map to grayscale

[0069] The depth camera is installed at a height of m meters above the ground, so the depth value range in the depth image is [0, m]. By default, the depth image is analyzed as a grayscale image, and the depth value is the grayscale value of the grayscale image. The entire image appears close to black. In order to facilitate the selection of feature points from the depth image, the depth image is processed to complete the mapping of depth values ​​to grayscale values, and a grayscale image with depth information is generated. The grayscale value range of the grayscale image is [0, 255], a total of 256 values. The basic idea of ​​mapping depth values ​​to grayscale values ​​is to map the different depth value ranges in the depth image with an interval of [0, m] to the interval [0, 255] to expand the difference between different pixel depths. Only the depth channel value is transformed. The depth value mapping principle is shown in formula (1), where z i is the grayscale value of the converted grayscale image, d i is the depth value in the original depth image, and i is the subscript of each pixel;

[0070]

[0071] b. Use grayscale image to register RGB image

[0072] The RGB image is matched with the pig region feature points in the grayscale image obtained in step S2(1)a by using feature point recognition and matching algorithms, and the registration coefficient [c x ,c y ], the RGB image is converted so that the pig body in the RGB image and the depth image coincide with each other, and the registered RGB image is obtained; the principle of RGB image conversion is shown in formula (2), where (x0, y0) is the coordinate of the RGB image before registration, (x, y) is the coordinate after registration, c x is the registration coefficient of the horizontal axis, c y is the registration coefficient of the ordinate;

[0073]

[0074] (2) Depth image conversion to point cloud

[0075] Get the internal parameters of the depth camera, focal length f x and f y , according to the camera parameter focal length, the data is converted from the image coordinate system to the world coordinate system to obtain the converted point cloud. The calculation formula is shown in formula (3):

[0076]

[0077] Among them, x, y, z are the point cloud coordinates, D is the depth value, and f x and f y is the focal length of the camera, x′ and y′ are the depth image coordinates, and the transformed point cloud contains the background and the pig body;

[0078] (3) Fusion of the registered RGB image and the converted point cloud

[0079] The data dimension of the converted point cloud in S2(2) is n×3, where n is the number of points in each point cloud and 3 is the data dimension of the coordinates (x, y, z) of each point. The data dimension of the registered RGB image in S2(1) is x×y×3, where x and y are the size, i.e., resolution, of the registered RGB image and 3 is the RGB value of each pixel, expressed as (r, g, b). Since the point cloud is converted from the depth image and the RGB image is registered from the depth image, the two images have the same coordinates x and y.

[0080] The registered RGB image is fused with the converted point cloud through the connection operation, which is denoted by the symbol ⊕. The fusion operation is defined as (x, y, z) ⊕ (r, g, b), which means connecting the coordinate point (x, y, z) of the point cloud with the RGB value (r, g, b). The fused data is still presented in the form of a point cloud. The dimension of the point cloud is n×6, where n is the number of points in the fused point cloud and 6 is the data dimension of each point. The data of each point after fusion is represented as (x, y, z, r, g, b).

[0081] S3: Fusion of original point cloud and fused point cloud

[0082] The fused point cloud obtained by S2 is fused with the original pig point cloud collected by the laser radar in step (1) of S1. The two parts of the point cloud include the pig body and the background respectively. First, the pig point cloud is segmented. The segmented pig body includes the left point cloud, the right point cloud and the rear hip point cloud of the pig respectively. Then, all the segmented point clouds are fused. The specific steps are as follows:

[0083] (1) Pig body point cloud segmentation

[0084] The point cloud consists of two parts: the pig body and the background. The KPFCNN method with a flexible convolution kernel is used to segment the pig body point cloud. The background point cloud is removed and the point cloud of the pig body is segmented. The nearest neighbor method is used for point upsampling and the corresponding layer feature propagation is used. The core operation of feature extraction is kernel point convolution. The feature of any point x is the sum of the features of all its neighboring points after kernel point convolution. The calculation is shown in formula (4):

[0085]

[0086] Among them, xi is the point x in the neighborhood space Nx, fi is the feature of the neighborhood point xi, and the function g is the kernel point convolution operation. In this model, a flexible convolution kernel is used, and Δ(x) is the offset of the kernel point. The model can generate the corresponding offset vector to change the spatial position of the kernel point according to the local structure in the point cloud, thereby generating a flexible convolution kernel that adapts to the local point cloud structure, and finally enables the model to extract features from pig point clouds of different shapes;

[0087] (2) Point cloud fusion

[0088] The segmented point cloud contains six parts, denoted as i, which are the point clouds of the pig's sides and the pig's rear hip collected by the laser radar in step S1 (1), and the point clouds of the pig's sides and the pig's rear hip fused by the RGB and depth images in step S2 (3). The point cloud dimension is n i ×6, i=1,2,3,4,5,6; where n i For the number of points in each point cloud, the six types of point clouds are directly connected, and the dimension of the fused point cloud is

[0089] S4: Fusion of point cloud and one-dimensional data

[0090] The dimension of the point cloud after fusion in step S3 The dimension of one-dimensional data is 1×3. First, the one-dimensional data is repeatedly expanded and the dimension becomes The dimension is The fused point cloud with dimension is The extended data of is used to connect the corresponding point features. The dimension of the final fused data is

[0091] S5: Establishment and training of sow body condition scoring model

[0092] Through the data fusion operations in steps S2, S3, and S4, the one-dimensional, two-dimensional, and three-dimensional data are fused, and finally the data layer fusion of the multimodal body condition assessment data is completed. The dimensions of the fused data are It is used as input to establish a body condition scoring model to complete the sow body condition scoring; the sow body condition scoring adopts a 5-level scoring standard. Since sows are generally fed with precision feeding and limited feeding, the body condition scores of sows in pig farms are concentrated at 1 and 2. As the score increases, the number of samples decreases significantly, and the number of sows with a score of 5 is already very small, showing a typical long-tail distribution feature. The sow body condition scoring adopts a 5-level scoring standard. In this application, a body condition scoring method based on category influence weighting (CIW) is proposed to address this problem:

[0093] The number of categories here is 5, corresponding to the 5 scoring levels of sows. The specific steps of CIW are as follows:

[0094] (1) Statistics of sample quantity of each category

[0095] Count the number of samples in each category i and get the number of samples n in the i-th category i , i=1,2,3,4,5;

[0096] (2) Forecast uncertainty assessment

[0097] The RepSurf model is used to predict samples of each category. For a single sample, the predicted entropy H is calculated as follows:

[0098] As shown in formula 5.

[0099]

[0100] where p i is the predicted probability of the i-th category. In order to calculate the average prediction uncertainty u for each category i i , the entropy of all samples belonging to this category needs to be averaged, and the calculation is shown in Formula 6. Among them, the number of samples in the i-th category n i , the entropy of each sample k is H k .

[0101]

[0102] (3) Category influence calculation

[0103] Combine the number of category samples and prediction uncertainty to calculate the influence I of the i-th category i , calculated as shown in formula (7):

[0104]

[0105] (4) Normalized weight

[0106] Normalize the influence I of all categories to get the final weight w of the i-th category i , calculated as shown in formula (8):

[0107]

[0108] (5) Application weight

[0109] During model training, the calculated category weights w are used i To adjust the loss function so that the model pays more attention to minority classes and classes with high prediction uncertainty;

[0110] A body condition scoring model (CIW-RepSurf) was constructed based on the CIW method and the basic framework of RepSurf. The overall structure of the model is as follows: Figure 2 As shown, it includes CIW weight calculation module, input module, feature extraction module and output module. First, the dimension of the data finally fused by S5 is The three-dimensional data is used as the input data of the model, and the spatial alignment and initialization of the fused data are completed through the input module. Then, the data initialized by the input module is passed to the feature extraction module, and the feature extraction module extracts features related to the body condition score from the fused data. At the same time, the weight calculation of different categories is completed through the CIW module. Specifically, the number of samples of the five categories with scores of 1-5 in the body condition score samples is counted respectively, and the weights w1, w2, w3, w4, and w5 of the five categories are calculated using the CIW method. Finally, the output features of the feature extraction module and the five weight values ​​of the CIW module are passed to the output module. The output module further aggregates the received features. At the same time, in the output module, the loss value is calculated using the weighted losses of different categories generated by the CIW module. The calculation of the loss value is shown in formula (9). Among them, L CIW (p,y) is the final loss function of the CIW-RepSurf model, L is the loss function before the weighted category loss, y is the five scoring categories, y∈{1,2,3,4,5}. p is the probability predicted by the model for different categories, then p=[p1,p2,p3,p4,p5] T , and p∈[0,1]. w is the loss weight calculated by CIW, w=[w1,w2,w3,w4,w5], w∈[0,1].

[0111] L CIW (p,y)=L(w×p,y) (9)

[0112] S6: Sow body condition scoring

[0113] During the model usage stage, RGB images, depth images and point clouds of the sow's two sides and rear buttocks are collected, and the sow's breed, parity and production stage data are obtained (input). The CIW-RepSurf model is used to generate the sow's body condition score, which ultimately meets the body condition scores of sows of multiple breeds and parities at different production stages. Accurate feeding and management are carried out according to the body condition scores of individual sows to maximize the sow's production capacity and obtain the maximum production benefits.

Claims

1. A method for comprehensive scoring of sow body condition by data layer fusion, the characteristics of which include Follow these steps: S1: Image acquisition and data acquisition The images and data that need to be acquired during the data acquisition phase include: (1) A 2D RGB image, a 3D depth image, and a 3D point cloud of the same position of the sow are acquired by using an RGB camera, a depth camera, and a laser radar respectively on the right side, the left side, and the back of the buttocks to realize image acquisition; wherein the RGB camera and the depth camera are set to the same resolution; (2) Obtain one-dimensional data on sow breed, parity, and reproductive stage; S2: RGB image and depth image are fused into point cloud (1) RGB image registration a. Convert the depth map to grayscale The depth camera is installed at a height of m meters above the ground, so the depth value range in the depth image is [0, m]. The depth image is processed to complete the mapping of depth value to grayscale value, and a grayscale image with depth information is generated. The grayscale value range of the grayscale image is [0, 255], a total of 256 values. The depth value mapping principle is shown in formula (1), where z i is the grayscale value of the converted grayscale image, d i is the depth value in the original depth image, and i is the subscript of each pixel; b. Use grayscale image to register RGB image The RGB image is matched with the pig region feature points in the grayscale image obtained in step S2(1)a by using feature point recognition and matching algorithms, and the registration coefficient [c x ,c y ], the RGB image is converted so that the pig body in the RGB image and the depth image coincide with each other, and the registered RGB image is obtained; the principle of RGB image conversion is shown in formula (2), where (x0, y0) is the coordinate of the RGB image before registration, (x, y) is the coordinate after registration, c x is the registration coefficient of the horizontal axis, c y is the registration coefficient of the ordinate; (2) Depth image conversion to point cloud Get the internal parameters of the depth camera, focal length f x and f y , transform the data from the image coordinate system to the world coordinate system to obtain the transformed point cloud. The calculation formula is shown in formula (3): Among them, x, y, z are the point cloud coordinates, D is the depth value, and f x and f y is the focal length of the camera, x′ and y′ are the depth image coordinates, and the transformed point cloud contains the background and the pig body; (3) Fusion of the registered RGB image and the converted point cloud The data dimension of the converted point cloud in S2(2) is n×3, where n is the number of points in each point cloud and 3 is the data dimension of the coordinates (x, y, z) of each point. The data dimension of the aligned RGB image in S2(1) is x×y×3, where x and y are the sizes of the aligned RGB image and 3 is the RGB value of each pixel, expressed as (r, g, b). The registered RGB image is fused with the converted point cloud through the connection operation, which is denoted by The fusion operation is defined as (x, y, z) ⊕ (r, g, b), which means connecting the coordinate point (x, y, z) of the point cloud with the RGB value (r, g, b). The fused data is still presented in the form of a point cloud. The dimension of the point cloud is n×6, where n is the number of points in the fused point cloud and 6 is the data dimension of each point. The data of each point after fusion is represented as (x, y, z, r, g, b). S3: Fusion of original point cloud and fused point cloud The fused point cloud obtained by S2 is fused with the original pig point cloud collected by the laser radar in step (1) of S1. The two parts of the point cloud include the pig body and the background respectively. First, the pig point cloud is segmented. The segmented pig body includes the left point cloud, the right point cloud and the rear hip point cloud of the pig respectively. Then, all the segmented point clouds are fused. The specific steps are as follows: (1) Pig body point cloud segmentation The point cloud consists of two parts: the pig body and the background. The KPFCNN method with a flexible convolution kernel is used to segment the pig body point cloud. The background point cloud is removed and the point cloud of the pig body is segmented. The nearest neighbor method is used for point upsampling and the corresponding layer feature propagation is used. The core operation of feature extraction is kernel point convolution. The feature of any point x is the sum of the features of all its neighboring points after kernel point convolution. The calculation is shown in formula (4): Among them, xi is the point x in the neighborhood space Nx, fi is the feature of the neighborhood point xi, and the function g is the kernel point convolution operation. In this model, a flexible convolution kernel is used, and Δ(x) is the offset of the kernel point. The model can generate the corresponding offset vector to change the spatial position of the kernel point according to the local structure in the point cloud, thereby generating a flexible convolution kernel that adapts to the local point cloud structure, and finally enables the model to extract features from pig point clouds of different shapes; (2) Point cloud fusion The segmented point cloud contains six parts, denoted as i, which are the point clouds of the pig's sides and the pig's rear hip collected by the laser radar in step S1 (1), and the point clouds of the pig's sides and the pig's rear hip fused by the RGB and depth images in step S2 (3). The point cloud dimension is n i ×6, i=1,2,3,4,5,6; where n i For the number of points in each point cloud, the six types of point clouds are directly connected, and the dimension of the fused point cloud is S4: Fusion of point cloud and one-dimensional data The dimension of the point cloud after fusion in step S3 The dimension of one-dimensional data is 1×3. First, the one-dimensional data is repeatedly expanded and the dimension becomes The dimension is The fused point cloud with dimension is The extended data of is used to connect the corresponding point features. The dimension of the final fused data is S5: Establishment and training of sow body condition scoring model Through the data fusion operations in steps S2, S3, and S4, the one-dimensional, two-dimensional, and three-dimensional data are fused, and finally the data layer fusion of the multimodal body condition assessment data is completed. The dimensions of the fused data are It is used as input to establish a body condition scoring model to complete the sow body condition scoring; the sow body condition scoring adopts a 5-level scoring standard and a body condition scoring method based on the weighted category influence: The number of categories here is 5, corresponding to the 5 scoring levels of sows. The specific steps of CIW are as follows: (1) Statistics of sample quantity of each category Count the number of samples in each category i and get the number of samples n in the i-th category i , i=1,2,3,4,5; (2) Forecast uncertainty assessment The RepSurf model is used to predict samples of each category. For a single sample, the predicted entropy H is calculated as shown in Formula 5: where p i is the predicted probability of the i-th category; in order to calculate the average prediction uncertainty u for each category i i , it is necessary to average the entropy of all samples belonging to this category, and the calculation is shown in Formula 6; where the number of samples in the i-th category n i , the entropy of each sample k is H k : (3) Category influence calculation Combine the number of category samples and prediction uncertainty to calculate the influence I of the i-th category i , calculated as shown in formula (7): (4) Normalized weight Normalize the influence I of all categories to get the final weight w of the i-th category i , calculated as shown in formula (8): (5) Application weight During model training, the calculated category weights w are used i To adjust the loss function; A body condition score model is constructed based on the CIW method and the basic architecture of RepSurf, including a CIW weight calculation module, an input module, a feature extraction module, and an output module. The CIW module completes the weight calculation of different categories, the input module completes the spatial alignment and initialization of the fused data, the feature extraction module extracts features related to the body condition score from the fused data, and the output module completes the prediction of the body condition score. In the output module, the loss value is calculated using the weighted loss of different categories generated by the CIW module. S6: Sow body condition scoring During the model usage phase, RGB images, depth images and point clouds of the sow’s two sides and rear hips are collected to obtain the sow’s breed, parity and production stage data. The CIW-RepSurf model is used to generate the sow’s body condition score, ultimately meeting the body condition score requirements for sows of multiple breeds, parities and different production stages.

2. A method for comprehensively scoring sow body condition by data layer fusion as claimed in claim 1, characterized in that According to the sow body condition score in step S6, accurate feeding and management are carried out.

Images