Sow weight estimation method based on multi-modal data feature layer fusion

Abstract

The invention relates to a multi-modal data feature layer fused sow weight estimation method, which relates to a detection method, and comprises the following steps: acquiring point clouds of sows at different angles through a sow point cloud acquisition device, and taking sow time sequence data such as variety, fetal number and reproduction stage and reconstructed sow real-time three-dimensional point clouds as input; a dual-branch network fused in a feature layer is established, meanwhile, deep features are automatically extracted from one-dimensional data and three-dimensional point clouds, time sequence data features and real-time data features are fused, and current body weight estimation of the sow and body weight curve estimation of the whole breeding stage are completed. Through the method, the accuracy of weight estimation is improved, the final output of the method is the weight real-time value of the current reproduction stage and the weight prediction value of the whole reproduction stage, and the real-time weight value is beneficial for obtaining the physical health condition of the sow at the current stage and adjusting the feeding strategy; therefore, the sows can reach the optimal health state and maintain the optimal production performance, and the minimum feed cost is maintained.

Description

Technical Field

[0001] The present invention relates to a detection method, and in particular to a sow weight estimation method based on multi-modal data feature layer fusion. Background Art

[0002] Weight can reflect the growth status, health status and reproductive performance of sows, and is an important indicator in health monitoring and feeding management. In terms of health monitoring, the weight of sows is an important indicator for maintaining appropriate nutritional and environmental levels. Deviation from the optimal growth state or sudden and substantial weight loss may be caused by improper feeding, malnutrition, environmental discomfort or infection. Monitoring of weight can timely understand the health status of sows and make timely adjustments to management strategies. In feeding management, on the one hand, feed costs are the main part of breeding costs, and maintaining an appropriate amount of feed supply can save feeding costs. On the other hand, at different stages of the birth process, sows also need to maintain a certain range of weight to achieve optimal production capacity. Therefore, according to the monitored weight, appropriate adjustment of feeding strategies to achieve precise feeding can achieve the purpose of weight control and cost savings.

[0003] The traditional way of weighing directly by a scale is time-consuming, laborious and easy to cause stress reactions in animals. Machine vision technology has the advantages of being intuitive and non-contact. In recent years, with the development of sensor technology and artificial intelligence technology, the research on estimating weight by obtaining animal appearance characteristics through machine vision has developed rapidly. For example, the publication number is CN113920453A, and the name is a method for estimating the body size and weight of pigs based on deep learning. It uses an RGB camera to collect RGB images of a single sow for weight estimation, but its accuracy is low. Point cloud can more accurately represent the three-dimensional body shape information of the pig body than its RGB image form. However, the current method of estimating weight through point cloud generally uses body size point recognition and point cloud computing to calculate body size, and further establishes a relationship model between body size and weight to estimate weight. For example, the publication number is CN113313833A, and the name is a method for estimating pig weight based on 3D vision technology. It converts the collected depth image into a point cloud, calculates the volume of the pig body on this basis, and calculates the weight through density, but the calculation is complex and the accuracy is low. In addition, the current machine vision-based methods generally make real-time estimates based on the current images of live pigs, without considering the combination of time-series weight data and real-time images to estimate the weight of live pigs; and the current weight of a sow is greatly correlated with the weight of different time periods. On the one hand, it is more correlated with the weight of the adjacent days. On the other hand, the weight of different sows will show some similar trends of change throughout the breeding stage. Therefore, the existing evaluation methods have low accuracy. To this end, the present application provides a sow weight estimation method that inputs multimodal data into a weight estimation model and fuses the data at the feature layer. Summary of the invention

[0004] The present invention aims to overcome the deficiencies of the prior art and provides a sow weight estimation method based on multimodal data feature layer fusion.

[0005] The sow weight estimation method based on multimodal data feature layer fusion of the present invention is achieved through the following steps:

[0006] S1: Data Collection

[0007] (1) 3D point cloud acquisition: Using a point cloud acquisition device, M sows in the barn are photographed at regular intervals every day to obtain two point clouds on both sides of each sow's body. Each point cloud can include a symmetrical half of the sow's body, including an image from head to tail;

[0008] (2) One-dimensional time series data acquisition: sow breed, parity, reproductive stage, and empty period weight data are obtained from the database;

[0009] S2: Data preprocessing

[0010] (1) Pig body point cloud segmentation: Segment the point cloud containing the background and the pig body captured in step S1, segment the point cloud of the pig body, remove the background point cloud, and output the three-dimensional point cloud of the pig body;

[0011] (2) Pig body point cloud reconstruction: Use the PACNet point cloud automatic registration algorithm to register the two point clouds in different directions of the pig body segmented in step S2 (1) to complete the pig body 3D point cloud reconstruction. The reconstructed point cloud contains the overall 3D contours of the pig body from the side and back.

[0012] S3: Establishment of weight estimation model: Using the one-dimensional time series data including sow breed, parity, breeding stage and initial weight of empty period in step S1 (2) and the three-dimensional point cloud of pig body reconstructed in step S2 (2) as input, the weight estimation model is used to estimate the weight of sows in the current breeding stage and predict the weight of sows in the entire breeding cycle;

[0013] The weight estimation model consists of three parts: input module, dual-branch feature extraction module and output module. The input module completes the preparation of input data and the alignment of point clouds, the dual-branch feature extraction module is used to learn the features of input data, and the output module converts the features into weight estimation values.

[0014] (I) Input module

[0015] The input data includes: one-dimensional time series data and reconstructed three-dimensional point cloud of pig body;

[0016] (1) One-dimensional time series data

[0017] The data is the data obtained in step (2) of S1, including sow breed, parity, breeding stage and initial weight during the empty period; the input time series length is fixed, the initial input data includes W0, and the input vector P = (p1, p2, p3);

[0018] W0 represents the initial body weight during the empty period;

[0019] p1 indicates the variety;

[0020] p2 indicates parity;

[0021] p3 represents the reproductive stage, which includes the empty period (0), breeding period (0), gestation period (1-114), delivery period (115), and lactation period (116-145);

[0022] (2) Reconstructed 3D point cloud of pig body

[0023] The data is the pig body 3D point cloud reconstructed in step S2(2). Each point cloud data contains n points. The coordinates of each point are world coordinates, expressed as (x, y, z). Therefore, the dimension of the input point cloud is n×3.

[0024] (II) Dual-branch feature extraction module

[0025] The dual-branch feature extraction module consists of two branches, namely the one-dimensional time series data feature extraction branch and the three-dimensional point cloud feature extraction branch. The two branches extract the time series features of sows by breed, parity and reproductive stage and the real-time features of sows' three-dimensional point clouds in parallel, and output 1×1024 features respectively. The two parts of features are connected as the final features of the module.

[0026] (1) One-dimensional time series data feature extraction branch

[0027] This branch is used to extract the characteristics of breed, parity, breeding stage and initial weight of empty period. The branch structure consists of 146 feature extraction modules with the same structure, corresponding to 146 breeding stages respectively. The network structure repeatedly iterates the same modules along the time series to achieve learning of time series data. Each module completes feature extraction through convolution operation (CNN), which specifically includes convolution layer, activation function and pooling layer. Each module can complete the weight estimation of sows in the current breeding stage.

[0028] The t-th module (t∈[1,2,...,146]) is used to extract the features of the t-th breeding stage. The input contains two parts. One part is the data P of the current breeding stage. t (p1, p2, p3), the other part of the input is the output weight characteristic value h of the previous breeding stage t-1 , the output feature h of the module t The feature extraction process is shown in formula (1);

[0029] h t =f(W in *P t +W t *h t-1 +b) (1)

[0030] Among them, h t represents the output weight value of the tth module, h t-1 represents the output weight value of the t-1th module, f represents the feature extraction function, W in is the network weight inside module t, W t is the network weight between module t-1 and module t, P t is the input of the t module, and b is the network bias;

[0031] The final output of the tth module is the estimated weight value h at the current breeding stage t The weight value calculated by weighting the initial weight W0 of the empty period, and introducing the residual structure, taking the initial weight W0 of the empty period as the influencing factor of the estimated result of this stage, the output O of the breeding stage t t The calculation is shown in formula (2);

[0032] O t =h t (1-W t )+W0*(W t ) (2)

[0033] Among them, h t is the weight estimation result directly output by the module before the residual calculation in stage t, W0 is the initial weight value in the empty period, and W t is a trainable weight parameter;

[0034] A fully connected layer is added at the end of the branch, and the output end is designed to have 1024 neurons. Finally, the timing branch outputs 1×1024 weight estimation features;

[0035] (2) 3D point cloud feature extraction branch

[0036] First, the geometric area for sampling is divided according to the contribution of different parts of the pig body to implement learning-based adaptive sampling (LBAS), which includes the following steps:

[0037] a. Divide the pig body point cloud into four regions: chest region (C), hip region (H), spine region (S) and other regions (O);

[0038] b. Obtaining the characteristic contribution values ​​of different regions c: During the training of the weight estimation model, in the last layer of the model, the characteristic values ​​of all points in the four regions are obtained; the average value method is used to calculate the mean of the characteristic values ​​of the points in each region as the contribution value of the region, as shown in formula (3):

[0039]

[0040] Where c is the contribution value of each region, f ij is the jth eigenvalue of the i-th point, n is the number of points in the area, and d is the characteristic dimension of each point;

[0041] c. Normalize the contribution value and calculate it as shown in formula (4):

[0042]

[0043] d. Calculate the sampling density d of different regions. The original sampling density is D. Multiply it by the contribution value to obtain the sampling density of four different regions. The calculation is shown in formula (5):

[0044] d i =w i ×D,i=1,2,3,4 (5)

[0045] Then, taking the sampled point cloud as input, the weight feature is extracted from the real-time 3D reconstructed pig point cloud. The branch processes the point cloud data in the form of a graph, regards the points in the point cloud as the vertices of the graph, and defines the edges by extracting the relationship between the vertices. In this branch, edge convolution is the main feature extraction module, which is responsible for extracting and generating edge features that describe the relationship between points and adjacent points. The specific process of feature extraction is defined as formula (6):

[0046]

[0047] Among them, f i Represents any point p in the input point cloud i and its characteristic neighborhood point p k The edge features between i is the set of points in the input point cloud, P k For point p i In the set of k nearest neighbor points among the input n points, h is the feature extraction function, max represents the maximum pooling operation, and all points in the input point cloud are point p i Neighborhood point, P k Any point in is defined as p j =(x j ,y j ,z j ), characterized by f j , P kThe corresponding k nearest neighbor distances d k It is calculated based on the Euclidean distance in the pig body point cloud feature space, as shown in formula (7):

[0048]

[0049] After being processed by each edge convolution module, vertex p i The features will be regenerated, and new k nearest neighbor points will be found in the feature space to reconstruct the feature graph. Feature extraction is achieved through a multi-layer perceptron. After being processed by the three-layer edge convolution module, the features extracted from different levels are connected, and the feature dimension is n×320. Subsequently, after being operated by the multi-layer perceptron, 1×1024 weight estimation features are obtained;

[0050] The output of the one-dimensional temporal branch is 1×1024, and the output of the three-dimensional real-time branch is also 1×1024. The two branch features are connected to generate a 1×2048 global feature, which is then passed to the output module;

[0051] (III) Output module

[0052] The output of the model consists of two parts, namely, the current weight estimate and the weight prediction for the entire reproductive cycle, which are represented by output module one and output module two respectively;

[0053] The output module 1 is mainly composed of a fully connected layer, which receives the 1*2048 features output from the dual-branch module, performs feature dimension reduction through the fully connected layer, and finally outputs the weight estimate; the output module contains three fully connected layers, whose output dimensions are {512, 256, 1}, and each linear layer is followed by a normalization operation BatchNorm, Dropout and a nonlinear function ReLu;

[0054] Output module 2 is a sequence of one-dimensional time series data feature extraction branches output on each time series module {O0, O1, ..., O 146}, respectively representing the estimated weight values ​​at different reproductive stages, and finally obtaining the estimated weight sequence of the individual in all reproductive stages;

[0055] S4: Training of weight estimation model

[0056] The weight data and corresponding point clouds of 1,000 sows of different breeds and parities at all reproductive stages were collected, including a total of 1,000 weight sequences of length 146, a total of 146,000 weight values ​​and corresponding point clouds, and 1,000 groups of sow breed and parity data were recorded; all data were divided into training set, validation set and test set in a ratio of 8:1:1; the point cloud segmentation model and weight estimation model were trained to determine all network parameters, and finally the optimal model was obtained;

[0057] S5: Weight estimation

[0058] During the model usage stage, the sow's breed, parity, current breeding stage and initial weight data of the empty-stomach period are obtained, the sow point cloud is collected in real time, the collected point cloud is input into the point cloud segmentation model, and after segmentation, it is input into the point cloud registration model to obtain the point cloud after three-dimensional reconstruction of the pig body. The sow's breed, parity, breeding stage data, initial weight of the empty-stomach period and the three-dimensional reconstructed point cloud of the pig body are input into the weight estimation module, and finally the current weight estimate and the weight prediction sequence of the entire production stage are output.

[0059] As a further improvement of the present invention, reasonable feeding is guided according to the estimated body weight in step S5.

[0060] The sow weight estimation method based on multimodal data feature layer fusion of the present invention has the following effects:

[0061] 1. This application adopts a weight estimation model with a dual-branch structure, which can extract multimodal data features from one-dimensional data and three-dimensional point clouds in parallel, complement the data features and image features, complete the fusion of multimodal data at the feature layer, and thus improve the accuracy of weight estimation;

[0062] 2. Learning-based point cloud adaptive sampling (LBAS) obtains the size of the feature values ​​of different regions during the model training process and uses them as weights to weight the sampling density. The sampling density is greater for regions that contribute more to the results, and the sampling density is smaller for regions that contribute less to the results. Since the number of points in the original point cloud is large, the efficiency of the model operation can be improved through the point cloud sampling strategy. The LBAS sampling strategy can make the sampling more accurate, improving efficiency while ensuring the accuracy of the results;

[0063] 3. Combine historical data with real-time images to estimate weight. On the one hand, the initial weight of sows in the empty period and the weight of sows in the reproductive stages that have been experienced are used as the basis of historical data to obtain the law of weight change of sows in different reproductive stages and predict the weight in the current and subsequent reproductive stages. On the other hand, weight is estimated based on the point cloud images collected in real time that can represent the body shape of the sow. The characteristics of data changes are combined with the characteristics of image representation to achieve weight estimation of sows in different reproductive stages, and at the same time, improve the accuracy of sow weight estimation;

[0064] 4. The accuracy of weight estimation through the reconstructed 3D point cloud of the pig body is higher. The collected original point cloud contains different angles of the two sides of the sow's body. The background interference is removed through point cloud segmentation, and then the two segmented pig point clouds are aligned to obtain the 3D reconstructed point cloud of the pig body. Compared with other image formats and single-angle point clouds, the 3D reconstructed point cloud can restore the sow's body shape more comprehensively and accurately, making the weight estimation more accurate;

[0065] 5. The final output of this method is the real-time weight value of the current reproductive stage and the predicted weight value of the entire reproductive stage. The real-time weight value is conducive to obtaining the current physical health status of the sow and the adjustment of the feeding strategy, so that the sow can achieve the best health state and maintain the best production performance, while maintaining the minimum feed cost. The predicted weight value of the entire reproductive stage is conducive to guiding reasonable production management, making appropriate guidance strategies and material preparations for sow production and piglet feeding, which is of great significance to sow feeding management and production management;

[0066] 6. This method can capture the changing characteristics of sows of different breeds and parities. Taking the breed and parity of the sow as the input of the model, the model can learn the different weight characteristics between breeds and the weight characteristics of different parities of the same sow based on a large amount of breed and parity data, ultimately making the method transferable in breed and parity. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the evaluation process of the present invention;

[0068] Figure 2 It is a model structure diagram of the present invention;

[0069] Figure 3 This is a regional diagram of the pig body point cloud division according to the present invention. DETAILED DESCRIPTION

[0070] The sow weight estimation method of the present invention based on multimodal data feature layer fusion is as follows: Figure 1 As shown, this is achieved through the following steps:

[0071] S1: Data Collection

[0072] (1) 3D point cloud acquisition: Using a point cloud acquisition device, M sows in the barn are photographed at regular intervals every day to obtain two point clouds on both sides of each sow's body. Each point cloud can include a symmetrical half of the sow's body, including an image from head to tail;

[0073] (2) One-dimensional time series data acquisition: sow breed, parity, reproductive stage, and empty period weight data are obtained from the database;

[0074] S2: Data preprocessing

[0075] (1) Pig body point cloud segmentation: Segment the point cloud containing the background and the pig body captured in step S1 to obtain the pig body point cloud, remove the interference of the background point cloud, and use the KPConv point cloud segmentation algorithm to take the point cloud with the background as input and output the pig body three-dimensional point cloud;

[0076] (2) Pig body point cloud reconstruction: Use the PACNet point cloud automatic registration algorithm to register the two point clouds in different directions of the pig body segmented in step S2 (1) to complete the pig body 3D point cloud reconstruction. The reconstructed point cloud contains the overall 3D contours of the pig body from the side and back.

[0077] S3: Establishment of weight estimation model: Using the one-dimensional time series data including sow breed, parity, breeding stage and initial weight of empty period in step S1 (2) and the three-dimensional point cloud of pig body reconstructed in step S2 (2) as input, the weight estimation model is used to estimate the weight of sows in the current breeding stage and predict the weight of sows in the entire breeding cycle;

[0078] The weight estimation model consists of three parts: input module, dual-branch feature extraction module and output module. The input module completes the preparation of input data and the alignment of point clouds, the dual-branch feature extraction module is used to learn the features of input data, and the output module converts the features into weight estimation values. The model structure is as follows: Figure 2 As shown;

[0079] (I) Input module

[0080] The input data includes: one-dimensional time series data and reconstructed three-dimensional point cloud of pig body;

[0081] (1) One-dimensional time series data

[0082] The data is the data obtained in step (2) of S1, including the sow breed, parity, breeding stage and initial weight during the empty period. The input time series length is fixed. The initial input data includes W0, and the input vector P = (p1, p2, p3);

[0083] W0 represents the initial body weight during the empty period;

[0084] p1 represents the breed, including Large White pig, Landrace pig, and hybrid pig (Landrace pig × Large White pig), p1∈{0,1,2};

[0085] p2 represents the parity, including: parous sows, multiparous sows are generally eliminated after 5 parities, p2∈{0,1,2,3,4,5};

[0086] p3 represents the reproductive stage, which includes the empty-skin period (0), breeding period (0), gestation period (1-114), parturition period (115), and lactation period (116-145). The gestation period of a sow, also known as the gestation period, is an average of 114 days. If a sow does not give birth after 114 days, it is represented by 114. p3∈{0,1,2,...,145}. The weight of the sow does not change much during the empty-skin period and breeding period, so it is represented by one moment; the gestation period is 114 days, and the weight will increase every day, so it is represented by 114 moments; the weight of the sow will decrease every day during the lactation period, so it is represented by 30 moments; that is, each day of the gestation period, parturition period, and lactation period corresponds to a reproductive stage and a feature module, for a total of 145 feature modules, while the empty-skin period and breeding period are one reproductive stage, corresponding to 1 feature module; making the entire reproductive stage a total of 146 feature modules;

[0087] (2) Reconstructed 3D point cloud of pig body

[0088] The data is the pig body 3D point cloud reconstructed in step S2(2). Each point cloud data contains n points. The coordinates of each point are world coordinates, expressed as (x, y, z). Therefore, the dimension of the input point cloud is n×3.

[0089] (II) Dual-branch feature extraction module

[0090] The dual-branch feature extraction module consists of two branches, namely the one-dimensional time series data feature extraction branch and the three-dimensional point cloud feature extraction branch. The two branches extract the time series features of sows by breed, parity and reproductive stage and the real-time features of sows' three-dimensional point clouds in parallel, and output 1×1024 features respectively. The two parts of features are connected as the final features of the module.

[0091] (1) One-dimensional time series data feature extraction branch

[0092] This branch is used to extract the characteristics of breed, parity, breeding stage and initial weight of the empty-stomach period. This time series branch has a memory function. The weight value of the current breeding stage is most affected by the previous moment. It is also related to the initial weight of the empty-stomach sow. The branch structure consists of 146 feature extraction modules with the same structure, corresponding to 146 breeding stages respectively. The network structure repeatedly iterates the same modules along the time series to achieve learning of time series data; each module completes feature extraction through convolution operation (CNN), which specifically includes convolution layer, activation function, and pooling layer. Each module can complete the weight estimation of sows in the current breeding stage;

[0093] The t-th module (t∈[1,2,...,146]) is used to extract the features of the t-th breeding stage. The input contains two parts. One part is the data P of the current breeding stage. t(p1, p2, p3), since the weight of the current breeding stage is most correlated with the weight of the previous breeding stage, in order to estimate the weight of the current breeding stage more accurately and quickly, the weight estimate of the previous stage is used as the input of this stage. The other part of the input is the output weight feature value h of the previous breeding stage t-1 , the output feature h of the module t The feature extraction process is shown in formula (1);

[0094] h t =f(W in *P t +W t *h t-1 +b) (1)

[0095] Among them, h t represents the output weight value of the tth module, h t-1 represents the output weight value of the t-1th module, f represents the feature extraction function, W in is the network weight inside module t, W t is the network weight between module t-1 and module t, P t is the input of the t module, and b is the network bias;

[0096] The final output of the tth module is the estimated weight value h at the current breeding stage t The weight value calculated by weighting the initial weight W0 of the empty period is designed to avoid forgetting the historical weight due to too many time series modules and too deep network structure, and to avoid the gradient disappearance caused by too long time series. Therefore, the residual structure is introduced, and the initial weight W0 of the empty period is used as the influencing factor of the estimated result of this stage. The output O of the breeding stage t t The calculation is shown in formula (2);

[0097] O t =h t (1-W t )+W0*(W t ) (2)

[0098] Among them, h t is the weight estimation result directly output by the module before the residual calculation in stage t, W0 is the initial weight value in the empty period, and W t is a trainable weighting parameter; as the reproductive stage deepens, the relationship between the initial weight and the final estimated weight becomes weaker, so a strategy of weighting by variable weights is proposed. However, the relationship between the specific reproductive stage and the initial weight is somewhat complex, so the weights are trained through the model to obtain a more accurate expression of the relationship.

[0099] A fully connected layer is added at the end of the branch, and the output end is designed to have 1024 neurons. Finally, the timing branch outputs 1×1024 weight estimation features;

[0100] (2) 3D point cloud feature extraction branch

[0101] First, due to the large number of points in the original point cloud, point cloud sampling is performed first to improve the efficiency of the model. Since point clouds from different parts of the pig body contribute differently to the final result. For example, the spine area can characterize the length, width and height characteristics of the pig body and contributes more. Therefore, the geometric area for sampling is divided according to the contribution of different parts of the pig body to achieve learning-based adaptive sampling (LBAS), which specifically includes the following steps:

[0102] a. Divide the pig body point cloud into four regions. Figure 3 As shown, chest area (C), hip area (H), spine area (S) and other areas (O);

[0103] b. Obtaining the characteristic contribution values ​​of different regions c: During the training of the weight estimation model, in the last layer of the model, the characteristic values ​​of all points in the four regions are obtained; the average value method is used to calculate the mean of the characteristic values ​​of the points in each region as the contribution value of the region, as shown in formula (3):

[0104]

[0105] Where c is the contribution value of each region, f ij is the jth eigenvalue of the i-th point, n is the number of points in the area, and d is the characteristic dimension of each point;

[0106] c. Normalize the contribution value and calculate it as shown in formula (4):

[0107]

[0108] d. Calculate the sampling density d of different regions. The original sampling density is D. Multiply it by the contribution value to obtain the sampling density of four different regions. The calculation is shown in formula (5):

[0109] d i =w i ×D,i=1,2,3,4 (5)

[0110] Then, taking the sampled point cloud as input, the weight feature is extracted from the real-time 3D reconstructed pig point cloud. The branch processes the point cloud data in the form of a graph, regards the points in the point cloud as the vertices of the graph, and defines the edges by extracting the relationship between the vertices. In this branch, edge convolution is the main feature extraction module, which is responsible for extracting and generating edge features that describe the relationship between points and adjacent points. The specific process of feature extraction is defined as formula (6):

[0111]

[0112] Among them, f i Represents any point p in the input point cloud i and its characteristic neighborhood point p k The edge features between i is the set of points in the input point cloud, P k For point p i In the set of k nearest neighbor points among the input n points, h is the feature extraction function, max represents the maximum pooling operation, and since no downsampling is performed, all points in the input point cloud are point p i Neighborhood point, P k Any point in is defined as p j =(x j ,y j ,z j ), characterized by f j , P k The corresponding k nearest neighbor distances d k It is calculated based on the Euclidean distance in the pig body point cloud feature space, as shown in formula (7):

[0113]

[0114] After being processed by each edge convolution module, vertex p i The features will be regenerated and new k nearest neighbors will be found in the feature space to reconstruct the feature map. Therefore, in the feature extraction branch of the feature neighborhood, the feature map is dynamically updated. Feature extraction is implemented through a multi-layer perceptron. Feature extraction is implemented through a multi-layer perceptron. After processing by the three-dimensional edge convolution module, the features extracted from different levels are connected, and the feature dimension is n×320. Subsequently, after the multi-layer perceptron operation, a 1×1024 weight estimation feature is obtained;

[0115] The output of the one-dimensional temporal branch is 1×1024, and the output of the three-dimensional real-time branch is also 1×1024. The two branch features are connected to generate a 1×2048 global feature, which is then passed to the output module;

[0116] (III) Output module

[0117] The output of the model consists of two parts, namely, the current weight estimate and the weight prediction for the entire reproductive cycle, which are represented by output module one and output module two respectively;

[0118] The output module 1 is mainly composed of a fully connected layer, which receives the 1*2048 features output from the dual-branch module, performs feature dimension reduction through the fully connected layer, and finally outputs the weight estimate; the output module contains three fully connected layers, whose output dimensions are {512, 256, 1}, and each linear layer is followed by a normalization operation BatchNorm, Dropout and a nonlinear function ReLu;

[0119] Output module 2 is a sequence of one-dimensional time series data feature extraction branches output on each time series module {O0, O1, ..., O 146}, respectively representing the estimated weight values ​​at different reproductive stages, and finally obtaining the estimated weight sequence of the individual in all reproductive stages;

[0120] S4: Training of weight estimation model

[0121] The weight data and corresponding point clouds of 1,000 sows of different breeds and parities at all reproductive stages were collected, including a total of 1,000 weight sequences of length 146, a total of 146,000 weight values ​​and corresponding point clouds, and 1,000 groups of sow breed and parity data were recorded; all data were divided into training set, validation set and test set in a ratio of 8:1:1; the point cloud segmentation model and weight estimation model were trained to determine all network parameters, and finally the optimal model was obtained;

[0122] S5: Weight estimation

[0123] In the model use stage, the breed, parity, current breeding stage and empty-stomach initial weight data of the sow are obtained, the sow point cloud is collected in real time, the collected point cloud is input into the point cloud segmentation model, and after segmentation, it is input into the point cloud registration model to obtain the point cloud after 3D reconstruction of the pig body. The breed, parity, breeding stage data of the sow, the empty-stomach initial weight and the 3D reconstruction point cloud of the pig body are input into the weight estimation module, and finally the current weight estimation value and the weight prediction sequence of the whole production stage are output;

[0124] S6: Guide production management and feeding management based on weight estimation results

[0125] Based on the current estimated weight, guide reasonable feeding to avoid feed waste caused by overfeeding, and avoid pregnancy and delivery risks caused by excessive obesity. Generate a growth curve based on the estimated weight sequence of the entire breeding stage, and make strategic adjustments and prepare materials and personnel for subsequent pregnancy management, delivery management, and piglet lactation management.

Claims

1. A sow weight estimation method based on multimodal data feature layer fusion is implemented by the following steps: S1: Data Collection (1) 3D point cloud acquisition: Using a point cloud acquisition device, M sows in the barn are photographed at regular intervals every day to obtain two point clouds on both sides of each sow's body. Each point cloud can include a symmetrical half of the sow's body, including an image from head to tail; (2) One-dimensional time series data acquisition: Obtain sow breed, parity, reproductive stage, and empty period weight data from the database; S2: Data preprocessing (1) Pig body point cloud segmentation: Segment the point cloud containing the background and the pig body captured in step S1, segment the point cloud of the pig body, remove the background point cloud, and output the three-dimensional point cloud of the pig body; (2) Pig body point cloud reconstruction: Use the PACNet point cloud automatic registration algorithm to register the two point clouds in different directions of the pig body segmented in step S2 (1) to complete the pig body 3D point cloud reconstruction. The reconstructed point cloud contains the overall 3D contours of the pig body from the side and back. S3: Establishment of weight estimation model: Using the one-dimensional time series data including sow breed, parity, breeding stage and initial weight of empty period in step S1 (2) and the three-dimensional point cloud of pig body reconstructed in step S2 (2) as input, the weight estimation model is used to estimate the weight of sows in the current breeding stage and predict the weight of sows in the entire breeding cycle; The weight estimation model consists of three parts: input module, dual-branch feature extraction module and output module; The input module completes the preparation of input data and the alignment of point clouds, the dual-branch feature extraction module is used to learn the features of input data, and the output module converts the features into weight estimates; (I) Input module The input data includes: one-dimensional time series data and reconstructed three-dimensional point cloud of pig body; (1) One-dimensional time series data The data is the data obtained in step (2) of S1, including sow breed, parity, breeding stage and initial weight during the empty period; the input time series length is fixed, the initial input data includes W0, and the input vector P = (p1, p2, p3); W0 represents the initial body weight during the empty period; p1 indicates the variety; p2 indicates parity; p3 represents the reproductive stage, which includes the empty period (0), breeding period (0), pregnancy period (1-114), delivery period (115), and lactation period (116-145), forming a total of 146 feature modules; (2) Reconstructed 3D point cloud of pig body The data is the pig body 3D point cloud reconstructed in step S2(2). Each point cloud data contains n points. The coordinates of each point are world coordinates, expressed as (x, y, z). Therefore, the dimension of the input point cloud is n×3. (II) Dual-branch feature extraction module The dual-branch feature extraction module consists of two branches, namely the one-dimensional time series data feature extraction branch and the three-dimensional point cloud feature extraction branch. The two branches extract the time series features of sows by breed, parity and reproductive stage and the real-time features of sows' three-dimensional point clouds in parallel, and output 1×1024 features respectively. The two parts of features are connected as the final features of the module. (1) One-dimensional time series data feature extraction branch This branch is used to extract the characteristics of breed, parity, breeding stage and initial weight of empty period. The branch structure consists of 146 feature extraction modules with the same structure, corresponding to 146 breeding stages respectively. The network structure repeatedly iterates the same modules along the time series to achieve learning of time series data. Each module completes feature extraction through convolution operations, including convolution layers, activation functions, and pooling layers. Each module can estimate the weight of sows in the current breeding stage. The t-th module (t∈[1,2,...,146]) is used to extract the features of the t-th breeding stage. The input contains two parts. One part is the data P of the current breeding stage. t (p1, p2, p3), the other part of the input is the output weight characteristic value h of the previous breeding stage t-1 , the output feature h of the module t The feature extraction process is shown in formula (1); h t =f(W in *P t +W t *h t-1 +b) (1) Among them, h t represents the output weight value of the tth module, h t-1 represents the output weight value of the t-1th module, f represents the feature extraction function, W in is the network weight inside module t, W t is the network weight between module t-1 and module t, P t is the input of the t module, and b is the network bias; The final output of the tth module is the estimated weight value h at the current breeding stage t The weight value calculated by weighting the initial weight W0 of the empty period, and introducing the residual structure, taking the initial weight W0 of the empty period as the influencing factor of the estimated result of this stage, the output O of the breeding stage t t The calculation is shown in formula (2); ABOUT t =h t (1-W t )+W0*(W t ) (2) Among them, h t is the weight estimation result directly output by the module before the residual calculation in stage t, W0 is the initial weight value in the empty period, and W t is a trainable weight parameter; A fully connected layer is added at the end of the branch, and the output end is designed to have 1024 neurons. Finally, the timing branch outputs 1×1024 weight estimation features; (2) 3D point cloud feature extraction branch First, the sampling geometric area is divided according to the contribution of different parts of the pig body to achieve learning-based adaptive sampling, which includes the following steps: a. Divide the pig body point cloud into four regions: chest region (C), hip region (H), spine region (S) and other regions (O); b. Obtaining the characteristic contribution values ​​of different regions c: During the training of the weight estimation model, in the last layer of the model, the characteristic values ​​of all points in the four regions are obtained; the average value method is used to calculate the mean of the characteristic values ​​of the points in each region as the contribution value of the region, as shown in formula (3): Where c is the contribution value of each region, f ij is the jth eigenvalue of the i-th point, n is the number of points in the area, and d is the characteristic dimension of each point; c. Normalize the contribution value and calculate it as shown in formula (4): d. Calculate the sampling density d of different regions. The original sampling density is D. Multiply it by the contribution value to obtain the sampling density of four different regions. The calculation is shown in formula (5): d i =w i ×D,i=1,2,3,4 (5) Then, taking the sampled point cloud as input, the weight feature is extracted from the real-time 3D reconstructed pig point cloud. The branch processes the point cloud data in the form of a graph, regards the points in the point cloud as the vertices of the graph, and defines the edges by extracting the relationship between the vertices. In this branch, edge convolution is the main feature extraction module, which is responsible for extracting and generating edge features that describe the relationship between points and adjacent points. The specific process of feature extraction is defined as formula (6): Among them, f i Represents any point p in the input point cloud i and its characteristic neighborhood point p k The edge features between i is the set of points in the input point cloud, P k For point p i In the set of k nearest neighbor points among the input n points, h is the feature extraction function, max represents the maximum pooling operation, and all points in the input point cloud are point p i Neighborhood point, P k Any point in is defined as p j =(x j ,y j ,z j ), characterized by f j , P k The corresponding k nearest neighbor distances d k It is calculated based on the Euclidean distance of the pig body point cloud feature space, as shown in formula (7): After being processed by each edge convolution module, vertex p i The features will be regenerated, and new k nearest neighbor points will be found in the feature space to reconstruct the feature graph. Feature extraction is achieved through a multi-layer perceptron. After being processed by the three-layer edge convolution module, the features extracted from different levels are connected, and the feature dimension is n×320. Subsequently, after being operated by the multi-layer perceptron, 1×1024 weight estimation features are obtained; The output of the one-dimensional temporal branch is 1×1024, and the output of the three-dimensional real-time branch is also 1×1024. The two branch features are connected to generate a 1×2048 global feature, which is then passed to the output module; (III) Output module The output of the model consists of two parts, namely, the current weight estimate and the weight prediction for the entire reproductive cycle, which are represented by output module one and output module two respectively; The output module 1 is mainly composed of a fully connected layer, which receives the 1*2048 features output from the dual-branch module, performs feature dimension reduction through the fully connected layer, and finally outputs the weight estimate; the output module contains three fully connected layers, whose output dimensions are {512, 256, 1}, and each linear layer is followed by a normalization operation BatchNorm, Dropout and a nonlinear function ReLu; Output module 2 is a sequence of one-dimensional time series data feature extraction branches, which are composed of the outputs of each time series module {O0, O1, ..., O 146 }, respectively representing the estimated weight values ​​at different reproductive stages, and finally obtaining the estimated weight sequence of the individual in all reproductive stages; S4: Training of weight estimation model The weight data and corresponding point clouds of 1,000 sows of different breeds and parities at all reproductive stages were collected, including a total of 1,000 weight sequences of length 146, a total of 146,000 weight values ​​and corresponding point clouds, and 1,000 groups of sow breed and parity data were recorded; all data were divided into training set, validation set and test set in a ratio of 8:1:1; the point cloud segmentation model and weight estimation model were trained to determine all network parameters, and finally the optimal model was obtained; S5: Weight estimation During the model usage stage, the sow's breed, parity, current breeding stage and initial weight data of the empty-stomach period are obtained, the sow point cloud is collected in real time, the collected point cloud is input into the point cloud segmentation model, and after segmentation, it is input into the point cloud registration model to obtain the point cloud after three-dimensional reconstruction of the pig body. The sow's breed, parity, breeding stage data, initial weight of the empty-stomach period and the three-dimensional reconstructed point cloud of the pig body are input into the weight estimation module, and finally the current weight estimate and the weight prediction sequence of the entire production stage are output.

2. A sow weight estimation method based on multimodal data feature layer fusion as claimed in claim 1, characterized in that According to the estimated weight in step S5, reasonable feeding is guided.

Images