A Method for Estimating Sows' Body Weight by Feature-Level Fusion of Multimodal Data

Abstract

A method for estimating the weight of sows by feature-level fusion of multimodal data according to the present invention relates to a detection method. Point clouds of sows from different angles are obtained through a sow point cloud acquisition device. Using sow time-series data such as breed, parity, and reproductive stage, and the reconstructed real-time three-dimensional point cloud of the sow as inputs, a dual-branch network that fuses at the feature level is established. Deep features are automatically extracted from one-dimensional data and three-dimensional point clouds at the same time, and the time-series data features and real-time data features are fused to complete the weight estimation of the sow at present and the weight curve prediction for the entire reproductive stage. Through this method, the accuracy of weight estimation is improved. The final output of this method is the real-time weight value at the current reproductive stage and the weight prediction value for the entire reproductive stage. The real-time weight value is conducive to obtaining the current physical health status of the sow and adjusting the feeding strategy, so that the sow can reach the best health status and maintain the best production performance, while maintaining the minimum feed cost.

Description

Technical Field

[0001] The present invention relates to a detection method, in particular to a method for estimating the weight of sows by fusing multi-modal data at the feature level. Background Art

[0002] Body weight can reflect the growth status, health status and reproductive performance of sows, and is an important indicator in the links of health monitoring and feeding management. In terms of health monitoring, the body weight of sows is an important indicator for maintaining an appropriate nutritional level and environmental level. Deviation from the optimal growth state or a sudden significant weight loss may be caused by improper feeding, malnutrition, environmental discomfort or disease infection. Monitoring the body weight can timely understand the health status of sows and make timely adjustments to management strategies. In feeding management, on the one hand, the feed cost is the main part of the breeding cost, and maintaining an appropriate feed supply can save breeding costs. On the other hand, in different production stages, sows also need to maintain a certain body weight range to achieve the best production capacity. Therefore, according to the monitored body weight, appropriately adjusting the feeding strategy to achieve precise feeding can achieve the purpose of controlling body weight and saving costs.

[0003] The traditional method of directly weighing by a weighing scale is time-consuming, laborious and prone to cause stress reactions in animals. Machine vision technology has the advantages of being intuitive and non-contact. In recent years, with the development of sensor technology and artificial intelligence technology, the research on estimating the body weight by obtaining the external characteristics of animals through machine vision means has developed rapidly. For example, the patent with the publication number CN113920453A and the name of a method for estimating the body size and weight of pigs based on deep learning estimates the body weight by collecting RGB images of a single sow through an RGB camera, but its accuracy is low. Point cloud can more accurately represent the three-dimensional body shape information of pigs compared with its RGB image form. However, currently, the method of estimating body weight by point cloud generally calculates the body size by means of body size point recognition and point cloud computing, and further establishes a relationship model between the body size and body weight for body weight estimation. For example, the patent with the publication number CN113313833A and the name of a method for estimating the body weight of pigs based on 3D vision technology converts the collected depth image into a point cloud, calculates the volume of the pig body on this basis, and calculates the body weight through density, but the calculation is complex and the accuracy is low. In addition, currently, the methods based on machine vision generally perform real-time estimation according to the current image of live pigs, and do not consider the method of combining time-series body weight data with real-time images for body weight estimation of live pigs; while there is a large correlation between the current body weight of sows and the body weight at different time-series periods. On the one hand, it is highly correlated with the body weight in the adjacent days. On the other hand, during the entire reproductive stage, the body weights of different sows will show some similar change trends. Therefore, the existing evaluation methods have low accuracy. For this reason, the present application provides a method for estimating the body weight of sows by inputting multi-modal data into a body weight estimation model and fusing data at the feature level. Summary of the Invention

[0004] The present invention aims to overcome the deficiencies of the prior art and provides a method for estimating the weight of sows by fusing multi-modal data at the feature level.

[0005] The method for estimating the weight of sows by fusing multi-modal data at the feature level of the present invention is realized through the following steps:

[0006] S1: Data collection

[0007] (1) Three-dimensional point cloud collection: Use a point cloud collection device to take pictures of M sows in the shed at a fixed time every day, and obtain two point clouds in the directions on both sides of each sow's body. Each point cloud can contain the symmetrical half of the sow's body, including the image from head to tail;

[0008] (2) Acquisition of one-dimensional time-series data: Obtain the breed, parity, reproductive stage, and non-pregnant weight data of sows from the database;

[0009] S2: Data preprocessing

[0010] (1) Pig body point cloud segmentation: Segment the point cloud containing the background and the pig body taken in step S1, segment out the point cloud of the pig body, remove the background point cloud, and output the three-dimensional point cloud of the pig body;

[0011] (2) Pig body point cloud reconstruction: Use the PACNet point cloud automatic registration algorithm to register the two point clouds in different directions of the pig body segmented in step (1) of S2, complete the three-dimensional point cloud reconstruction of the pig body, and the reconstructed point cloud contains the overall three-dimensional contours of the side and back of the pig body;

[0012] S3: Establishment of a weight estimation model: Use the one-dimensional time-series data containing the breed, parity, reproductive stage, and initial non-pregnant weight of sows in step (2) of S1 and the three-dimensional point cloud of the pig body reconstructed in step (2) of S2 as inputs, and estimate the weight of sows in the current reproductive stage and predict the weight of sows throughout the reproductive cycle through the weight estimation model;

[0013] The weight estimation model includes three parts: an input module, a double-branch feature extraction module, and an output module; among them, the input module completes the preparation of input data and the alignment of point clouds, the double-branch feature extraction module is used to learn the features of input data, and the output module converts the features into weight estimates;

[0014] (I) Input module

[0015] The input data includes: one-dimensional time-series data and the three-dimensional point cloud of the pig body reconstructed;

[0016] (1) One-dimensional time-series data

[0017] This data is the data obtained in step (2) of S1, including sow breed, parity, reproductive stage, and initial weight during anestrus; the input time series length is fixed, the initial input data includes W0, and the input vector P = (p1, p2, p3);

[0018] W0 represents the initial weight during anestrus;

[0019] p1 represents the breed;

[0020] p2 represents the parity;

[0021] p3 represents the reproductive stage, and the reproductive stage includes anestrus (0), breeding period (0), gestation period (1 - 114), parturition period (115), lactation period (116 - 145);

[0022] (2) The reconstructed three-dimensional point cloud of the pig body

[0023] This data is the reconstructed three-dimensional point cloud of the pig body in step S2(2). The data of each point cloud contains n points, and the coordinates of each point are world coordinates, expressed as (x, y, z). Therefore, the dimension of the input point cloud is n×3;

[0024] (II) Dual-branch feature extraction module

[0025] The dual-branch feature extraction module contains two branches, namely the one-dimensional time series data feature extraction branch and the three-dimensional point cloud feature extraction branch, which parallelly extract the sow time series features of breed, parity, and reproductive stage and the real-time features of the sow three-dimensional point cloud, and output features of 1×1024 respectively. The two parts of features are concatenated as the final feature of this module;

[0026] (1) One-dimensional time series data feature extraction branch

[0027] This branch is used to extract the features of breed, parity, reproductive stage, and initial weight during anestrus. The branch structure consists of 146 feature extraction modules with the same structure, corresponding to 146 reproductive stages respectively. The network structure iteratively repeats the same module along the time series to realize the learning of time series data; among them, each module completes feature extraction through convolutional operations (CNN), specifically including a convolutional layer, an activation function, and a pooling layer. Each module can complete the weight estimation of sows in the current reproductive stage;

[0028] The t-th module (t ∈ [1, 2,..., 146]) is used to extract the features of the t-th reproductive stage. The input includes two parts. One part is the data P t (p1, p2, p3) of the current reproductive stage, and the other part of the input is the output weight feature value h t-1 of the previous reproductive stage. The feature extraction process of the output feature h t of the module is shown in formula (1);

[0029] h t = f(W in *P t + W t *h t-1 + b)(1)

[0030] Among them, h t represents the output body weight value of the t-th module, h t-1 represents the output body weight value of the (t - 1)-th module, f represents the feature extraction function, W in is the network weight inside the t-th module, W t is the network weight between the (t - 1)-th module and the t-th module, P t is the input of the t-th module, and b is the network bias;

[0031] The final output of the t-th module is the estimated body weight value h t of the current breeding stage weighted with the initial body weight W0 during the anestrus period. At the same time, a residual structure is introduced, and the initial body weight W0 during the anestrus period is used as an influencing factor for the estimation result of this stage. The output O t of the breeding stage t is calculated as shown in formula (2);

[0032] O t = h t (1 - W t ) + W0 * (W t )(2)

[0033] Among them, h t is the body weight estimation result directly output by the module before the residual calculation in the t-th stage, W0 is the initial body weight value during the anestrus period, and W t is the trainable weighting parameter;

[0034] A fully connected layer is added at the end of the branch, and the output end is designed with 1024 neurons. Finally, this temporal branch outputs a 1×1024 body weight prediction feature;

[0035] (2) 3D point cloud feature extraction branch

[0036] First, divide the sampled geometric regions according to the contribution degrees of different parts of the pig body to achieve learning-based adaptive sampling (LBAS), which specifically includes the following steps:

[0037] a. Divide the pig body point cloud into four regions, namely the chest region (C), the hip region (H), the spine region (S), and the other region (O);

[0038] b. Obtain the feature contribution value c of different regions: During the training of the body weight prediction model, at the last layer of the model, obtain the feature values of all points in the four regions; use the average method to calculate the average of the feature values of the points in each region as the contribution value of this region, and the calculation is as shown in formula (3):

[0039]

[0040] where c is the contribution value of each region, f ij is the j-th feature value of the i-th point, n is the number of points in this region, and d is the feature dimension of each point;

[0041] c. Normalize the contribution value, and the calculation is as shown in formula (4):

[0042]

[0043] d. Calculate the sampling density d of different regions. The original sampling density is D, and multiply it by the contribution value to obtain the sampling densities of the four different regions. The calculation is as shown in formula (5):

[0044] d i = w i × D, i = 1, 2, 3, 4 (5)

[0045] Then, using the sampled point cloud as the input, extract the body weight features from the point cloud of the pig body after real-time 3D reconstruction. The branch processes the point cloud data in the form of a graph, regards the points in the point cloud as the vertices of the graph, and defines the edges by extracting the relationships between the vertices. In this branch, edge convolution is used as the main feature extraction module, which is responsible for extracting and generating the edge features describing the relationship between points and adjacent points. The specific process of feature extraction is defined as shown in formula (6):

[0046]

[0047] where, f i represents the edge feature between any point p i in the input point cloud and its feature neighborhood point p k , P i is the set of points in the input point cloud, P k is the set of the k nearest neighbor points of point p i in the input n points, h is the feature extraction function, max represents the max pooling operation, all points in the input point cloud are the neighborhood points of point p i , and any point in P k is defined as p j = (x j , y j , z j ), the feature is f j , P kThe distances d of the corresponding k nearest neighbor points k It is calculated according to the Euclidean distance in the feature space of the pig body point cloud, as shown in formula (7):

[0048]

[0049] After being processed by each edge convolution module, vertex p i will regenerate features and find new k nearest neighbor points in the feature space, thus reconstructing the feature map. Feature extraction is achieved through a multi-layer perceptron. After being processed by three edge convolution modules, the features extracted at different levels are connected, and the feature dimension is n×320. Subsequently, through the operation of the multi-layer perceptron, a weight prediction feature of 1×1024 is obtained;

[0050] The output of the one-dimensional time series branch is 1×1024, and the output of the three-dimensional real-time branch is also 1×1024. The features of the two branches are connected to generate a 1×2048 global feature, which is then passed to the output module;

[0051] (III) Output module

[0052] The output of this model includes two parts, namely the current weight estimate value and the weight prediction for the entire reproductive cycle, which are represented by output module 1 and output module 2 respectively;

[0053] Output module 1 is mainly composed of fully connected layers. This module receives the 1*2048 features output from the dual-branch module and performs feature dimensionality reduction through the fully connected layers, and finally outputs the weight prediction value; The output module contains three fully connected layers, and their output dimensions are {512, 256, 1}, and each linear layer is followed by a normalization operation BatchNorm, Dropout, and a non-linear function ReLu;

[0054] Output module 2 is a sequence {O0, O1,..., O 146} composed of the outputs of the one-dimensional time series data feature extraction branch on each time series module, which respectively represent the weight prediction values at different reproductive stages, and finally obtain the weight prediction sequence of the entire reproductive stage of the individual;

[0055] S4: Training of the weight prediction model

[0056] Collect the weight data and corresponding point clouds of 1000 sows of different breeds and parities during the entire reproductive stage, which altogether contain 1000 weight sequences with a length of 146, a total of 146,000 weight values and corresponding point clouds, and record the breed and parity data of 1000 groups of sows; Divide all the data into a training set, a validation set, and a test set according to the ratio of 8:1:1; Train the point cloud segmentation model and the weight prediction model to determine all network parameters, and finally obtain the optimal model;

[0057] S5: Body weight estimation

[0058] In the model usage stage, obtain the breed, parity, current reproductive stage, and initial body weight data during anestrus of the sow. Real-time collect the point cloud of the sow. The collected point cloud is input into the point cloud segmentation model. After segmentation, it is input into the point cloud registration model to obtain the point cloud after the three-dimensional reconstruction of the pig body. Input the breed, parity, reproductive stage data, initial body weight during anestrus, and the three-dimensional reconstruction point cloud of the pig body into the body weight estimation module, and finally output the current body weight estimation value and the body weight prediction sequence for the entire production stage.

[0059] As a further improvement of the present invention, according to the body weight estimation value in step S5, guide reasonable feeding.

[0060] The method for estimating the body weight of sows by fusing multi-modal data feature layers of the present invention has the following effects:

[0061] 1. The body weight estimation model adopted in this application has a double-branch structure, which can parallelly extract the features of multi-modal data from one-dimensional data and three-dimensional point clouds, complement the data features and image features, complete the fusion of multi-modal data at the feature layer, and thus improve the accuracy of body weight estimation;

[0062] 2. Based on learning-based point cloud adaptive sampling (LBAS), obtain the feature value sizes of different regions during the model training process, and use them as weights to weight the sampling density. The regions that contribute more to the result have a larger sampling density, and the regions that contribute less to the result have a smaller sampling density. Since the number of points in the original point cloud is large, the point cloud sampling strategy can improve the operation efficiency of the model. Adopting the LBAS sampling strategy can make the sampling more accurate, ensuring the accuracy of the result while improving the efficiency;

[0063] 3. Combine historical data with real-time images for body weight estimation. On the one hand, use the initial body weight of sows during anestrus and the body weights of previous reproductive stages as the basis of historical data to obtain the rules of body weight changes of sows in different reproductive stages and predict the body weights of the current and subsequent reproductive stages. On the other hand, estimate the body weight according to the point cloud images that can characterize the body shape of the sow collected in real time. Combine the characteristics of data changes with the characteristics represented by images to achieve the body weight estimation of sows in different reproductive stages, and at the same time, improve the accuracy of sow body weight estimation;

[0064] 4. The accuracy of body weight estimation using the three-dimensional reconstructed point cloud of the pig body is higher. The collected original point cloud includes different angles on both sides of the sow's body. Remove the background interference through point cloud segmentation, and then register the two pig body point clouds obtained by segmentation to obtain the three-dimensional reconstructed point cloud of the pig body. Compared with other image formats and point clouds from a single angle, the three-dimensional reconstructed point cloud can restore the body shape of the sow more comprehensively and accurately, making the body weight estimation more accurate;

[0065] 5. The final output of this method is the real-time weight value at the current breeding stage and the predicted weight value for the entire breeding stage. The real-time weight value is beneficial for obtaining the current physical health status of the sow and adjusting the feeding strategy, so as to enable the sow to reach the best health status and maintain the best production performance, while maintaining the minimum feed cost. The predicted weight value for the entire breeding stage is beneficial for guiding reasonable production management, making appropriate guiding strategies and material preparations for sow production and piglet feeding, and has important significance for the feeding management and production management of sows;

[0066] 6. This method can capture the changing characteristics of sows of different breeds and parities. Using the breed and parity of the sow as the input of the model, the model can learn the different weight characteristics between breeds and the weight characteristics of the same sow in different parities based on a large amount of breed and parity data, and finally make this method transferable in terms of breed and parity. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the evaluation process of the present invention;

[0068] Figure 2 It is a structural diagram of the model of the present invention;

[0069] Figure 3 It is a diagram of the divided regions of the pig body point cloud of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0070] The method for estimating the weight of sows by fusing multi-modal data feature layers of the present invention, as Figure 1 shown, is implemented through the following steps:

[0071] S1: Data collection

[0072] (1) Three-dimensional point cloud collection: Using a point cloud collection device, M sows in the shed are photographed regularly every day to obtain two point clouds in the directions on both sides of each sow's body. Each point cloud can contain the symmetrical half of the sow's body, including the image from head to tail;

[0073] (2) Acquisition of one-dimensional time-series data: Obtain the breed, parity, breeding stage, and non-pregnant weight data of the sow from the database;

[0074] S2: Data preprocessing

[0075] (1) Pig body point cloud segmentation: Segment the point cloud containing the background and the pig body photographed in step S1, segment out the point cloud of the pig body, remove the interference of the background point cloud, and use the KPConv point cloud segmentation algorithm. Taking the point cloud with the background as the input, output the three-dimensional point cloud of the pig body;

[0076] (2) Pig body point cloud reconstruction: Using the PACNet point cloud automatic registration algorithm, register the two point clouds of different directions of the pig body segmented in step (1) of S2 to complete the three-dimensional point cloud reconstruction of the pig body. The reconstructed point cloud contains the overall three-dimensional contours of the side and back of the pig body;

[0077] S3: Establishment of the body weight estimation model: Using the one-dimensional time-series data including sow breed, parity, reproductive stage, and initial body weight during anestrus in step (2) of S1 and the three-dimensional point cloud of the reconstructed pig body in step (2) of S2 as inputs, estimate the body weight of sows in the current reproductive stage and predict the body weight of sows throughout the reproductive cycle through the body weight estimation model;

[0078] The body weight estimation model consists of three parts: an input module, a dual-branch feature extraction module, and an output module; among them, the input module completes the preparation of input data and the alignment of point clouds, the dual-branch feature extraction module is used to learn the features of input data, and the output module converts the features into body weight estimates. The model structure is as Figure 2 shown;

[0079] (I) Input module

[0080] The input data includes: one-dimensional time-series data and the three-dimensional point cloud of the reconstructed pig body;

[0081] (1) One-dimensional time-series data

[0082] This data is the data obtained in step (2) of S1, including sow breed, parity, reproductive stage, and initial body weight during anestrus. The input time series length is fixed. The initial input data includes W0, and the input vector P = (p1, p2, p3);

[0083] W0 represents the initial body weight during anestrus;

[0084] p1 represents the breed, including: Large White Pig, Landrace Pig, Two-way Crossbred Pig (Landrace Pig × Large White Pig), p1 ∈ {0, 1, 2};

[0085] p2 represents the parity, including: first-parity sow, sows in subsequent parities are generally culled after 5 parities, p2 ∈ {0, 1, 2, 3, 4, 5};

[0086] p3 represents the reproductive stage, which includes the anestrus period (0), breeding period (0), gestation period (1 - 114), parturition period (115), and lactation period (116 - 145). Among them, the gestation period of sows, also known as the pregnancy period, is on average 114 days. If the sow does not give birth after 114 days, it is represented by 114. p3 ∈ {0, 1, 2,..., 145}. The body weight change is generally small during the anestrus and breeding periods, so it is represented by one moment; the gestation period is 114 days, and the body weight increases every day, so it is represented by 114 moments; the body weight decreases every day during the lactation period, so it is represented by 30 moments; that is, each day of the gestation, parturition, and lactation periods corresponds to one reproductive stage and also one feature module, with a total of 145 feature modules. The anestrus and breeding periods are one reproductive stage, corresponding to 1 feature module; making the entire reproductive stage have a total of 146 feature modules;

[0087] (2) The reconstructed three-dimensional point cloud of the pig body

[0088] This data is the three-dimensional point cloud of the pig body reconstructed in step S2(2). The data of each point cloud contains n points, and the coordinates of each point are world coordinates, represented as (x, y, z). Therefore, the dimension of the input point cloud is n×3;

[0089] (2) Dual-branch feature extraction module

[0090] The dual-branch feature extraction module contains two branches, namely the one-dimensional time-series data feature extraction branch and the three-dimensional point cloud feature extraction branch. It parallelly extracts the time-series features of the breed, parity, and reproductive stage of sows and the real-time features of the three-dimensional point cloud of sows, and outputs features of 1×1024 respectively. The two parts of the features are concatenated as the final feature of this module;

[0091] (1) One-dimensional time-series data feature extraction branch

[0092] This branch is used to extract the features of breed, parity, reproductive stage, and the initial body weight during anestrus. This time-series branch has a memory function. The body weight value at the current reproductive stage is most affected by the previous moment, and there is also an association with the initial body weight of sows during anestrus. The branch structure consists of 146 feature extraction modules with the same structure, corresponding to 146 reproductive stages respectively. The network structure iteratively repeats the same module along the time series to achieve the learning of time-series data; each module completes feature extraction through convolutional operations (CNN), specifically including a convolutional layer, an activation function, and a pooling layer. Each module can complete the body weight estimation of sows at the current reproductive stage;

[0093] The t-th module (t ∈ [1, 2,..., 146]) is used to extract the features of the t-th reproductive stage. The input contains two parts. One part is the data P of the current reproductive stage t(p1, p2, p3), since the weight in the current breeding stage has the greatest correlation with the weight in the previous breeding stage, in order to estimate the weight in the current breeding stage more accurately and quickly, the estimated weight value of the previous stage is used as the input of this stage. Another part of the input is the output weight feature value h of the previous breeding stage t-1 , the output feature h of the module t The feature extraction process is shown in formula (1);

[0094] h t = f(W in *P t + W t *h t-1 + b) (1)

[0095] Among them, h t represents the output weight value of the t-th module, h t-1 represents the output weight value of the (t - 1)-th module, f represents the feature extraction function, W in is the network weight inside the t-th module, W t is the network weight between the (t - 1)-th module and the t-th module, P t is the input of the t-th module, and b is the network bias;

[0096] The final output of the t-th module is the estimated weight value h of the current breeding stage t weight value calculated by weighted calculation with the initial weight W0 in the anestrus period. The purpose of this design is to avoid forgetting the historical weight due to too many timing modules and too deep network structure, and at the same time to avoid the vanishing gradient caused by too long timing. Therefore, a residual structure is introduced, and the initial weight W0 in the anestrus period is used as an influencing factor for the estimation result of this stage. The output O of the breeding stage t t is calculated as shown in formula (2);

[0097] O t = h t (1 - W t ) + W0 * (W t ) (2)

[0098] Among them, h t is the weight estimation result directly output by the module before the residual calculation in the t-th stage, W0 is the initial weight value in the anestrus period, and W t is the trainable weighting parameter; as the network deepens backward in the breeding stage, the relationship between the initial weight and the final estimated weight changes from strong to weak. Therefore, a strategy of weighted calculation through variable weights is proposed. However, the relationship between the specific breeding stage and the initial weight has a certain complexity. Therefore, the model is used to train the weights to obtain a more accurate relationship expression.

[0099] Add a fully connected layer at the end of the branch, with the output designed to have 1024 neurons. Finally, the time series branch outputs weight prediction features of 1×1024.

[0100] (2) 3D point cloud feature extraction branch

[0101] First, since the number of points in the original point cloud is large, to improve the efficiency of model operation, point cloud sampling is first performed. Since the point clouds of different parts of the pig body contribute differently to the final result. For example, the spine area can characterize the length, width, and height features of the pig body and contributes more. Therefore, the geometric regions for sampling are divided according to the contribution degrees of different parts of the pig body to achieve learning-based adaptive sampling (LBAS), which specifically includes the following steps:

[0102] a. Divide the pig body point cloud into four regions, as shown in Figure 3 Figure, chest region (C), hip region (H), spine region (S), and other region (O);

[0103] b. Obtain the feature contribution value c of different regions: During the training process of the weight prediction model, at the last layer of the model, obtain the feature values of all points in the four regions; use the average method to calculate the average of the feature values of the points in each region as the contribution value of this region, and the calculation is as shown in formula (3):

[0104]

[0105] where c is the contribution value of each region, f ij is the j-th feature value of the i-th point, n is the number of points in this region, and d is the feature dimension of each point;

[0106] c. Normalize the contribution value, and the calculation is as shown in formula (4):

[0107]

[0108] d. Calculate the sampling density d of different regions. The original sampling density is D, and multiply it by the contribution value to obtain the sampling densities of the four different regions. The calculation is as shown in formula (5):

[0109] d i = w i ×D, i = 1, 2, 3, 4 (5)

[0110] Then, using the sampled point cloud as the input, the body weight feature is extracted from the point cloud of the pig body after real-time three-dimensional reconstruction. The branch processes the point cloud data in the form of a graph, regarding the points in the point cloud as the vertices of the graph, and defining the edges by extracting the relationships between the vertices. In this branch, edge convolution serves as the main feature extraction module, responsible for extracting and generating the edge features that describe the relationship between a point and its adjacent points. The specific process of feature extraction is defined as in formula (6):

[0111]

[0112] Among them, f i represents the edge feature between any point p i in the input point cloud and its feature neighborhood point p k . P i is the set of points in the input point cloud, P k is the set of the k-nearest neighbor points of point p i among the input n points. h is the feature extraction function, max represents the max pooling operation. Since no downsampling is performed, all points in the input point cloud are the neighborhood points of point p i . Any point in P k is defined as p j =(x j , y j , z j ). The feature is f j . The k nearest neighbor point distances d k corresponding to P k are calculated according to the Euclidean distance in the feature space of the pig body point cloud, as shown in formula (7):

[0113]

[0114] After being processed by each edge convolution module, vertex p i will regenerate the feature and find new k-nearest neighbor points in the feature space, thus reconstructing the feature map. Therefore, in the feature extraction branch of the feature neighborhood, the feature map is dynamically updated. Feature extraction is implemented through a multi-layer perceptron. Feature extraction is implemented through a multi-layer perceptron. After being processed by three edge convolution modules, the features extracted at different levels are concatenated, and the feature dimension is n×320. Subsequently, through the operation of the multi-layer perceptron, a body weight prediction feature of 1×1024 is obtained;

[0115] The output of the one-dimensional time series branch is 1×1024, and the output of the three-dimensional real-time branch is also 1×1024. The features of the two branches are concatenated to generate a 1×2048 global feature, which is then passed to the output module;

[0116] (III) Output module

[0117] The output of the model consists of two parts, namely the current weight estimate and the weight prediction for the entire reproductive cycle, which are represented by output module 1 and output module 2 respectively;

[0118] Output module 1 is mainly composed of fully connected layers. This module receives the 1*2048 features output from the double-branch module, and performs feature dimensionality reduction through the fully connected layers, and finally outputs the weight prediction value. The output module contains three fully connected layers, and their output dimensions are {512, 256, 1} respectively. And after each linear layer, there are normalization operations BatchNorm, Dropout and non-linear function ReLu;

[0119] Output module 2 is a sequence {O0, O1,..., O 146} composed of the outputs of the one-dimensional time series data feature extraction branch on each time series module, which respectively represent the weight prediction values at different reproductive stages, and finally obtain the weight prediction sequence of the entire reproductive stage of the individual;

[0120] S4: Training of the weight prediction model

[0121] Collect the weight data and corresponding point clouds of 1000 sows of different breeds and parities during the entire reproductive stage, which altogether contain 1000 weight sequences with a length of 146, a total of 146,000 weight values and the corresponding point clouds, and record the breed and parity data of 1000 groups of sows; Divide all the data into a training set, a validation set and a test set according to the ratio of 8:1:1; Train the point cloud segmentation model and the weight prediction model to determine all network parameters, and finally obtain the optimal model;

[0122] S5: Weight prediction

[0123] During the model usage stage, obtain the breed, parity, current reproductive stage and initial weight data during the non-pregnant period of the sow, collect the sow point cloud in real time, input the collected point cloud into the point cloud segmentation model, and after segmentation, input it into the point cloud registration model to obtain the point cloud after the three-dimensional reconstruction of the pig body. Input the breed, parity, reproductive stage data, initial weight during the non-pregnant period and the three-dimensional reconstruction point cloud of the pig body of the sow into the weight prediction module, and finally output the current weight estimate and the weight prediction sequence of the entire production stage;

[0124] S6: Guide production management and feeding management according to the weight prediction results

[0125] According to the current weight prediction value, guide reasonable feeding to avoid feed waste caused by overfeeding, and at the same time avoid the risks of pregnancy and childbirth brought by over-obesity. Generate a growth curve according to the predicted weight sequence of the entire reproductive stage, and make strategy adjustments and material and personnel preparations for subsequent pregnancy management, childbirth management and piglet nursing management, etc.

Claims

1. A sow weight estimation method based on multimodal data feature layer fusion is implemented by the following steps: S1: Data Collection (1) 3D point cloud acquisition: Using a point cloud acquisition device, M sows in the barn are photographed at regular intervals every day to obtain two point clouds on both sides of each sow's body. Each point cloud can include a symmetrical half of the sow's body, including an image from head to tail; (2) One-dimensional time series data acquisition: Obtain sow breed, parity, reproductive stage, and empty period weight data from the database; S2: Data preprocessing (1) Pig body point cloud segmentation: Segment the point cloud containing the background and the pig body captured in step S1, segment the point cloud of the pig body, remove the background point cloud, and output the three-dimensional point cloud of the pig body; (2) Pig body point cloud reconstruction: Use the PACNet point cloud automatic registration algorithm to register the two point clouds in different directions of the pig body segmented in step S2 (1) to complete the pig body 3D point cloud reconstruction. The reconstructed point cloud contains the overall 3D contours of the pig body from the side and back. S3: Establishment of weight estimation model: Using the one-dimensional time series data including sow breed, parity, breeding stage and initial weight of empty period in step S1 (2) and the three-dimensional point cloud of pig body reconstructed in step S2 (2) as input, the weight estimation model is used to estimate the weight of sows in the current breeding stage and predict the weight of sows in the entire breeding cycle; The weight estimation model consists of three parts: input module, dual-branch feature extraction module and output module; The input module completes the preparation of input data and the alignment of point clouds, the dual-branch feature extraction module is used to learn the features of input data, and the output module converts the features into weight estimates; (I) Input module The input data includes: one-dimensional time series data and reconstructed three-dimensional point cloud of pig body; (1) One-dimensional time series data The data is the data obtained in step (2) of S1, including sow breed, parity, breeding stage and initial weight during the empty period; the input time series length is fixed, the initial input data includes W0, and the input vector P = (p1, p2, p3); W0 represents the initial body weight during the empty period; p1 indicates the variety; p2 indicates parity; p3 represents the reproductive stage, which includes the empty period (0), breeding period (0), pregnancy period (1-114), delivery period (115), and lactation period (116-145), forming a total of 146 feature modules; (2) Reconstructed 3D point cloud of pig body The data is the pig body 3D point cloud reconstructed in step S2(2). Each point cloud data contains n points. The coordinates of each point are world coordinates, expressed as (x, y, z). Therefore, the dimension of the input point cloud is n×3. (II) Dual-branch feature extraction module The dual-branch feature extraction module consists of two branches, namely the one-dimensional time series data feature extraction branch and the three-dimensional point cloud feature extraction branch. The two branches extract the time series features of sows by breed, parity and reproductive stage and the real-time features of sows' three-dimensional point clouds in parallel, and output 1×1024 features respectively. The two parts of features are connected as the final features of the module. (1) One-dimensional time series data feature extraction branch This branch is used to extract the characteristics of breed, parity, breeding stage and initial weight of empty period. The branch structure consists of 146 feature extraction modules with the same structure, corresponding to 146 breeding stages respectively. The network structure repeatedly iterates the same modules along the time series to achieve learning of time series data. Each module completes feature extraction through convolution operations, including convolution layers, activation functions, and pooling layers. Each module can estimate the weight of sows in the current breeding stage. The t-th module (t∈[1,2,...,146]) is used to extract the features of the t-th breeding stage. The input contains two parts. One part is the data P of the current breeding stage. t (p1, p2, p3), the other part of the input is the output weight characteristic value h of the previous breeding stage t-1 , the output feature h of the module t The feature extraction process is shown in formula (1); h t =f(W in *P t +W t *h t-1 +b) (1) Among them, h t represents the output weight value of the tth module, h t-1 represents the output weight value of the t-1th module, f represents the feature extraction function, W in is the network weight inside module t, W t is the network weight between module t-1 and module t, P t is the input of the t module, and b is the network bias; The final output of the tth module is the estimated weight value h at the current breeding stage t The weight value calculated by weighting the initial weight W0 of the empty period, and introducing the residual structure, taking the initial weight W0 of the empty period as the influencing factor of the estimated result of this stage, the output O of the breeding stage t t The calculation is shown in formula (2); ABOUT t =h t (1-W t )+W0*(W t ) (2) Among them, h t is the weight estimation result directly output by the module before the residual calculation in stage t, W0 is the initial weight value in the empty period, and W t is a trainable weight parameter; A fully connected layer is added at the end of the branch, and the output end is designed to have 1024 neurons. Finally, the timing branch outputs 1×1024 weight estimation features; (2) 3D point cloud feature extraction branch First, the sampling geometric area is divided according to the contribution of different parts of the pig body to achieve learning-based adaptive sampling, which includes the following steps: a. Divide the pig body point cloud into four regions: chest region (C), hip region (H), spine region (S) and other regions (O); b. Obtaining the characteristic contribution values ​​of different regions c: During the training of the weight estimation model, in the last layer of the model, the characteristic values ​​of all points in the four regions are obtained; the average value method is used to calculate the mean of the characteristic values ​​of the points in each region as the contribution value of the region, as shown in formula (3): Where c is the contribution value of each region, f ij is the jth eigenvalue of the i-th point, n is the number of points in the area, and d is the characteristic dimension of each point; c. Normalize the contribution value and calculate it as shown in formula (4): d. Calculate the sampling density d of different regions. The original sampling density is D. Multiply it by the contribution value to obtain the sampling density of four different regions. The calculation is shown in formula (5): d i =w i ×D,i=1,2,3,4 (5) Then, taking the sampled point cloud as input, the weight feature is extracted from the real-time 3D reconstructed pig point cloud. The branch processes the point cloud data in the form of a graph, regards the points in the point cloud as the vertices of the graph, and defines the edges by extracting the relationship between the vertices. In this branch, edge convolution is the main feature extraction module, which is responsible for extracting and generating edge features that describe the relationship between points and adjacent points. The specific process of feature extraction is defined as formula (6): Among them, f i Represents any point p in the input point cloud i and its characteristic neighborhood point p k The edge features between i is the set of points in the input point cloud, P k For point p i In the set of k nearest neighbor points among the input n points, h is the feature extraction function, max represents the maximum pooling operation, and all points in the input point cloud are point p i Neighborhood point, P k Any point in is defined as p j =(x j ,y j ,z j ), characterized by f j , P k The corresponding k nearest neighbor distances d k It is calculated based on the Euclidean distance of the pig body point cloud feature space, as shown in formula (7): After being processed by each edge convolution module, vertex p i The features will be regenerated, and new k nearest neighbor points will be found in the feature space to reconstruct the feature graph. Feature extraction is achieved through a multi-layer perceptron. After being processed by the three-layer edge convolution module, the features extracted from different levels are connected, and the feature dimension is n×320. Subsequently, after being operated by the multi-layer perceptron, 1×1024 weight estimation features are obtained; The output of the one-dimensional temporal branch is 1×1024, and the output of the three-dimensional real-time branch is also 1×1024. The two branch features are connected to generate a 1×2048 global feature, which is then passed to the output module; (III) Output module The output of the model consists of two parts, namely, the current weight estimate and the weight prediction for the entire reproductive cycle, which are represented by output module one and output module two respectively; The output module 1 is mainly composed of a fully connected layer, which receives the 1*2048 features output from the dual-branch module, performs feature dimension reduction through the fully connected layer, and finally outputs the weight estimate; the output module contains three fully connected layers, whose output dimensions are {512, 256, 1}, and each linear layer is followed by a normalization operation BatchNorm, Dropout and a nonlinear function ReLu; Output module 2 is a sequence of one-dimensional time series data feature extraction branches output on each time series module {O0, O1, ..., O 146 }, respectively representing the estimated weight values ​​at different reproductive stages, and finally obtaining the estimated weight sequence of the individual in all reproductive stages; S4: Training of weight estimation model The weight data and corresponding point clouds of 1,000 sows of different breeds and parities at all reproductive stages were collected, including a total of 1,000 weight sequences of length 146, a total of 146,000 weight values ​​and corresponding point clouds, and 1,000 groups of sow breed and parity data were recorded; all data were divided into training set, validation set and test set in a ratio of 8:1:1; the point cloud segmentation model and weight estimation model were trained to determine all network parameters, and finally the optimal model was obtained; S5: Weight estimation During the model usage stage, the sow's breed, parity, current breeding stage and initial weight data of the empty-stomach period are obtained, the sow point cloud is collected in real time, the collected point cloud is input into the point cloud segmentation model, and after segmentation, it is input into the point cloud registration model to obtain the point cloud after three-dimensional reconstruction of the pig body. The sow's breed, parity, breeding stage data, initial weight of the empty-stomach period and the three-dimensional reconstructed point cloud of the pig body are input into the weight estimation module, and finally the current weight estimate and the weight prediction sequence of the entire production stage are output.

2. A sow weight estimation method based on multimodal data feature layer fusion as claimed in claim 1, characterized in that According to the estimated weight in step S5, reasonable feeding is guided.

Images