Multi-modal live pig weight measuring method and system based on attention fusion

Abstract

The invention provides a multi-modal pig weight measurement method and system based on attention fusion, and the method comprises the steps: S1, data collection: employing a depth camera to synchronously obtain an RGB image and a depth image of a pig, and converting the depth image into point cloud data; s2, multi-modal feature extraction: extracting 2D features from the RGB image, and extracting 3D features from the point cloud data; s3, feature fusion: performing cross-modal feature fusion based on an attention mechanism on the extracted 2D features and 3D features to obtain attention fusion features; and S4, MLP regression prediction: inputting the attention fusion features into an MLP weight regression prediction model, and predicting the weight of the pig through the MLP weight regression prediction model. According to the method, the 2D features and the 3D features are fused, so that the adaptability of the system in a complex scene is enhanced, and the accuracy of weight prediction can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data recognition and measurement, and in particular to a multimodal pig weight measurement method and system based on attention fusion. Background Art

[0002] China is the world's largest pig producer and pork consumer, ranking first in both pig farming scale and pork consumption. Pig weight is a crucial indicator for assessing pig growth and development, and a key factor in breeding, reproductive performance assessment, and precision feeding. However, traditional pig weight measurement methods suffer from numerous issues, primarily low efficiency, large errors, and a heavy reliance on manual operation. The limitations of traditional methods are particularly evident in large-scale pig farms, making them unable to meet the demands of modern animal husbandry for efficient, precise, and automated management.

[0003] In recent years, the rapid development of deep learning technology in object detection and image recognition has ushered in new breakthroughs in precision agriculture. By combining depth cameras and deep learning algorithms, pig weight estimation no longer relies on traditional manual measurement, but can now be achieved through fast and accurate automated measurement.

[0004] Existing depth camera technology used in pig weight estimation primarily involves using monocular, binocular, and depth cameras to collect data and then estimate weight. While monocular cameras offer low cost for weight estimation, their limitations in depth information acquisition, environmental adaptability, and computational complexity limit their widespread application in practical farming environments, particularly in scenarios requiring high precision and real-time performance. While binocular cameras can provide a certain level of depth information, they also face challenges such as high calibration requirements, high computational complexity, and poor environmental adaptability. These shortcomings also limit their practical application. In contrast, single depth cameras offer some improvement in depth information provision, enabling more accurate depth images. However, due to the relatively sparse point cloud images, weight estimation using sparse point cloud data remains suboptimal. Therefore, while depth cameras have somewhat offset the shortcomings of monocular and binocular cameras, they still face challenges in practical applications.

[0005] Furthermore, many current deep learning-based weight measurement solutions typically obtain body measurement data and then fit it to weight information to predict weight. Since the acquisition of body measurement data itself contains errors, the errors are amplified when fitting it through a deep learning model, ultimately leading to insufficient accuracy in weight measurement results.

[0006] Therefore, the existing measurement methods based on traditional image processing and deep learning algorithms still have a lot of room for improvement, especially in the environment of large-scale breeding and high-precision requirements. The existing technology still has the defects of poor adaptability to complex scenes and insufficient prediction accuracy. Summary of the Invention

[0007] The purpose of the present invention is to address the defects in the existing technology and provide a multimodal pig weight measurement method and system based on attention fusion, enhance the adaptability of the system in complex scenarios, and improve the progress and robustness of pig weight measurement.

[0008] To achieve the above object, the present invention adopts the following technical solutions: A multimodal pig weight measurement method based on attention fusion, comprising the following steps: S1. Data acquisition: Use a depth camera to synchronously acquire RGB images and depth images of pigs, and convert the depth images into point cloud data; S2, multimodal feature extraction: extract 2D features from RGB images and 3D features from point cloud data; S3, feature fusion: The extracted 2D features and 3D features are fused across modalities based on the attention mechanism to obtain attention fusion features; S4, MLP regression prediction: Input the attention fusion features into the MLP weight regression prediction model, and predict the weight of the pigs through the MLP weight regression prediction model; The MLP weight regression prediction model is obtained by the following method: S401, using the attention fusion feature as input and the pig's weight as output, establish an MLP weight regression prediction model; S402. Take several pigs as samples and generate a piece of sample data (A, W) for each sample, where A is the attention fusion feature obtained by the methods of S1 to S3, and W is the actual weight of the pig; S403: Using sample data to perform supervised training on the MLP weight regression prediction model.

[0009] Furthermore, the S1 also includes, after converting the depth image into point cloud data, performing bicubic interpolation enhancement processing on the point cloud data to insert new points in the sparse point cloud and improve the point cloud density.

[0010] Furthermore, in S2, an improved convolutional neural network based on the ResNet50 network is used to extract 2D features from RGB images; The improved convolutional neural network is improved on the basis of the ResNet50 network. The improvements include: adding a residual attention module after the Conv1 layer of the original ResNet50 network; replacing the 3×3 standard convolution of the original ResNet50 network with a depthwise separable convolution; and introducing a spatial pyramid pooling layer after the last convolution layer of the original ResNet50 network to output an RGB global feature vector representing 2D features.

[0011] Furthermore, in S2, the PointNet++ network is used to extract 3D features from point cloud data.

[0012] Furthermore, in S3, the method of cross-modal feature fusion of the extracted 2D features and 3D features based on the attention mechanism is as follows: 2D features and 3D features are spliced ​​along the channel dimension to generate a joint feature vector; the channel importance of the spliced ​​features is recalibrated through the SE module, and the channel weights are generated after compressing the global information; the weighted features are input into the cross-attention layer, with 2D features as queries and 3D features as key-values, the cross-modal correlation is calculated, a fusion weight map is generated, and finally the attention fusion features are output.

[0013] Furthermore, it also includes S5, evaluation record: setting a weight threshold range according to the individual situation of the pig, and evaluating the effectiveness of the weight of the pig obtained in S4; when the obtained weight is within the weight threshold range, recording the weight information; when the obtained weight is outside the weight threshold range, discarding the weight information and re-measuring the weight of the pig from S1.

[0014] A multimodal pig weight measurement system based on attention fusion, used to implement the above-mentioned multimodal pig weight measurement method based on attention fusion, comprising: Depth camera, used to obtain RGB images and depth images of pigs; Point cloud data conversion module, used to convert depth images into point cloud data; 2D feature extraction module, used to extract 2D features from RGB images; 3D feature extraction module, used to extract 3D features from point cloud data; The cross-modal attention fusion module is used to fuse the extracted 2D features and 3D features based on the attention mechanism to obtain attention fusion features; The MLP weight regression prediction model is used to predict the weight information of pigs based on attention fusion features.

[0015] Furthermore, it also includes an evaluation and recording module, which is used to set a weight threshold range according to the individual situation of the pig, and to evaluate the effectiveness of the pig's weight obtained by the MLP weight regression prediction model; when the obtained weight is within the weight threshold range, the weight information is recorded; when the obtained weight is outside the weight threshold range, the weight information is discarded.

[0016] This invention uses a non-contact weight measurement method, completely avoiding the stress response caused by traditional contact measurement on animals, and can better meet the needs of efficient and automated monitoring in large-scale farms. By fusing 2D and 3D features to form a fusion feature with joint spatial and texture representation capabilities, the invention enhances the system's adaptability in complex scenarios and provides highly robust and discriminative input data for the subsequent MLP weight regression prediction model, significantly improving the accuracy of weight prediction.

[0017] The present invention has the excellent characteristics of high precision, strong robustness and explainability, and is widely applicable to scenarios such as smart farming, on-site transactions, and health monitoring. It provides a solid data foundation and decision-making support for pig weight estimation and subsequent volume and health assessment, and can effectively assist in the intelligent upgrade of the farming industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of the steps of a multimodal pig weight measurement method based on attention fusion provided by an embodiment of the present invention.

[0019] Figure 2 This is a data processing flow chart of a multimodal pig weight measurement method based on attention fusion provided by an embodiment of the present invention.

[0020] Figure 3 This is a data processing flow chart for cross-modal feature fusion in S3 of an embodiment of the present invention.

[0021] Figure 4 This is a functional module diagram of a multimodal pig weight measurement system based on attention fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a multimodal pig weight measurement method based on attention fusion, comprising the following steps: S1. Data acquisition: Use a depth camera to synchronously acquire RGB images and depth images of pigs, and convert the depth images into point cloud data.

[0024] S2, multimodal feature extraction: extract 2D features from RGB images and 3D features from point cloud data; S3, feature fusion: The extracted 2D features and 3D features are fused across modalities based on the attention mechanism to obtain attention fusion features; S4, MLP (Multilayer Perceptron) regression prediction: The attention fusion features are input into the MLP weight regression prediction model, and the pig weight is predicted by the MLP weight regression prediction model; The MLP weight regression prediction model is obtained by the following method: S401, using the attention fusion feature as input and the pig's weight as output, establishing an MLP weight regression prediction model based on a multi-layer perceptron; S402. Take several pigs as samples and generate a piece of sample data (A, W) for each sample, where A is the attention fusion feature obtained by the methods of S1 to S3, and W is the actual weight of the pig; S403: Using sample data to perform supervised training on the MLP weight regression prediction model.

[0025] Specifically, in S1, to facilitate data collection, this embodiment installed a fence bracket within the piggery's passageway to ensure that each pig could pass individually. An Intel D435i depth camera was installed directly above the passageway, approximately 0.5 to 2 meters from the pigs, to capture RGB and depth images of the pigs. The D435i depth camera uses Time of Flight (TOF) ranging technology to acquire depth information. Its operating principle is that an infrared transmitter actively emits continuous infrared light pulses, while a depth sensor receives these reflected infrared light pulses from an object and calculates the distance from the object to the sensor based on the time of flight of the light pulses. This technology has strong anti-interference capabilities, reducing dependence on external lighting conditions. Furthermore, through higher depth fidelity and noise suppression, it can more accurately capture 3D information of objects. Calibration of the D435i depth camera enables more accurate depth images to be obtained. The captured depth images are converted into point cloud data, and noise removal further improves the quality of the point cloud, providing reliable data support for subsequent weight measurements.

[0026] Compared with the existing technology, one of the key improvements of the present invention is that before extracting 3D features, the depth image is converted into point cloud data, and then the 3D features are extracted from the point cloud data. In the existing technology, 3D features are extracted directly from the depth image. This improvement helps to improve the data dimension and spatial structure integrity, and improve the robustness and generalization ability of 3D feature extraction. Three-dimensional point cloud data can fully capture the spatial layout of various postures of pigs such as standing, lying on their side, and turning around. At the same time, it can also use three-dimensional redundant information to infer occluded areas, reconstruct missing points after interpolation, and improve the completeness of weight estimation. Therefore, first converting the depth image into point cloud data and then extracting 3D features from the point cloud data has better adaptability to complex environments and postures, and improves the visualization and interpretability of the data.

[0027] As an improvement, the S1 also includes, after converting the depth image into point cloud data, performing bicubic interpolation enhancement processing on the point cloud data to insert new points in the sparse point cloud and increase the point cloud density, thereby obtaining finer three-dimensional surface details.

[0028] The specific method of bicubic interpolation enhancement is: for the target point ( x , y ) 16 known points in a 4×4 neighborhood around it, and use the cubic polynomial kernel function to calculate the interpolation coefficients c ij , and get the depth or attribute value of the new point according to the following formula:

[0029] in, z ( x , y ) represents the target point ( x , y ) is obtained by interpolation of the depth or attribute value, i Indicates the target point ( x , y ) The relative position index of the known points in the 4×4 neighborhood around it in the X-axis direction, j Indicates the target point ( x , y ) The relative position index of the known points in the 4×4 neighborhood around it in the Y-axis direction, φ ( x - i ) represents the interpolation coefficient of the cubic polynomial kernel function in the X-axis direction, φ ( y - j ) represents the interpolation coefficient of the cubic polynomial kernel function in the Y-axis direction.

[0030] By performing bicubic interpolation on point cloud data, the density of the point cloud can be significantly increased, making it easier to capture subtle concave and convex features such as the pig's ears and tail, and enhancing the model's understanding of small 3D structures. The density-enhanced point cloud can smoothly reconstruct the 3D surfaces of key areas such as the pig's back and abdomen, avoiding the discontinuities and blurring common in 2D depth map reconstruction. Furthermore, bicubic interpolation not only increases the number of point clouds but also generates a variety of neighborhood distribution patterns, enriching the distribution of training samples and improving the model's generalization ability.

[0031] Furthermore, in S2, this embodiment uses an improved convolutional neural network based on the ResNet50 network to extract 2D features from RGB images. Specifically, this improved convolutional neural network is derived from the ResNet50 network. Through a series of improvements and optimizations, it can maintain feature expression capabilities while reducing computational complexity, thereby more efficiently extracting pig texture and morphological features from RGB images. The improvements include: adding a residual attention block after the Conv1 layer of the original ResNet50 network to enhance the capture of edge and local texture features (such as pig backfat wrinkles); replacing the original ResNet50 network's 3×3 standard convolution with a depthwise separable convolution, which reduces computational complexity by over 60% while maintaining accuracy; and introducing a spatial pyramid pooling layer (SPP) after the last convolution layer of the original ResNet50 network to output a multi-scale fused RGB global feature vector.

[0032] Furthermore, in S2, this embodiment uses the PointNet++ network to extract 3D features from point cloud data. This 3D feature extraction method specifically includes: Based on the hierarchical feature extraction architecture of the PointNet++ network, a multi-scale grouping (MSG) strategy is employed to sample point clouds within radii of 0.1m, 0.3m, and 0.5m to extract local geometric details (pig backfat data); hierarchical farthest point sampling (FPS) is used to gradually expand the receptive field, ultimately generating a three-dimensional structural feature vector that includes body length and volume; and when transferring features at each layer, the low-dimensional features of the previous layer are cross-resolution concatenated with the high-dimensional features of the current layer to preserve fine-grained information.

[0033] Through the improved convolutional neural network, the 2D features that can be extracted from the RGB image are a 1024-dimensional (1×1×1024) RGB global feature vector; through the PointNet++ network, the 3D features that can be extracted from the point cloud data are a 1024-dimensional (1×1×1024) three-dimensional structural feature vector.

[0034] Furthermore, if Figure 3 As shown, in S3, the method of performing cross-modal feature fusion based on the attention mechanism on the extracted 2D features and 3D features includes: S301, Self-Attention Extraction: Feed the 2D features (1024-dimensional RGB global feature vector) and 3D features (1024-dimensional three-dimensional structural feature vector) into two independent Transformer encoders to calculate their own self-attention features, thereby extracting their respective high-dimensional expressions and enhancing the structured semantic representation within the modality; S302, cross-attention fusion: The 2D features, 3D features and the calculation results of S301 are input into a cross-attention fusion module, which includes multiple Transformer decoder layers. The multiple Transformer decoder layers achieve deep semantic fusion through an odd-even alternating structure: in the odd layers, 3D features are used as queries (Query), and 2D features are used as keys (Key) and values ​​(Value) to achieve the perception and alignment of spatial structure to image appearance information; in the even layers, the roles are swapped, and 2D features are used as queries (Query), and 3D features are used as keys (Key) and values ​​(Value) to guide the image information to enhance the perception of spatial geometry.

[0035] Through the above steps, the final fusion feature obtained is a 2048-dimensional attention fusion vector, which is input into the trained MLP weight regression prediction model to perform weight prediction.

[0036] The cross-attention fusion module in this paper uses an alternating cross-attention mechanism to fully integrate multimodal features, enabling complementary enhancement of two-dimensional and three-dimensional features at multiple semantic levels. Experimental results show that this fusion strategy effectively improves the model's global perception and detailed characterization of the target while maintaining structural integrity.

[0037] In the task of pig weight estimation, the strategy of multimodal feature fusion has a significant impact on model performance. Currently, common fusion methods mainly include feature concatenation and cross-modal attention fusion. The traditional concatenation method concatenates the feature vectors extracted from RGB images and point cloud data in dimensions and then inputs them into the MLP for weight regression prediction. This method has a simple structure, low computational overhead, and is easy to deploy quickly. However, this strategy only statically fuses multimodal features and fails to effectively establish the mutual relationship and semantic dependency between modalities, thereby limiting the model's expressive power and generalization capabilities. In particular, when one modality fails (such as image blur or sparse point cloud), performance degrades significantly.

[0038] In contrast, the cross-modal attention fusion method adopted by the present invention enables dynamic and in-depth interaction between 2D features and 3D features by introducing a cross-attention mechanism in the decoder under the Transformer framework. The model can adaptively adjust the attention weight of each modality according to the feature content of the current input to achieve more refined multimodal feature fusion. By performing cross-modal feature fusion based on the attention mechanism on 2D features and 3D features, the present invention not only retains the complete description of the spatial structure of the three-dimensional point cloud, but also takes into account the supplement of the appearance details by the RGB texture. It not only improves the perception of key body parts, but also enhances the robustness of the model to environmental changes and individual differences of different pigs, and can achieve high-precision and strong robustness of pig weight estimation.

[0039] Furthermore, the MLP weight regression prediction model is built on an MLP (Multilayer Perceptron) architecture, consisting of a sequentially connected input layer, hidden layer, and output layer. The input layer receives a fused 2048-dimensional feature vector; the hidden layer employs a two-layer structure: a 512-dimensional fully connected layer in the first layer and a 128-dimensional fully connected layer in the second layer. Each layer is followed by a BatchNorm, ReLU activation function, and a Dropout layer with a Dropout rate set to 0.3 to prevent overfitting. The output layer is a 1-dimensional linear regression unit that outputs the predicted weight value in kilograms.

[0040] In S403, when training the MLP weight regression prediction model, supervised training is performed using the mean square error loss function, the optimizer selects AdamW, the initial learning rate is set to 1e-4, and a weight decay of 1e-5 is introduced to prevent overfitting.

[0041] As an improvement, after the pig's weight is predicted in S4, S5, evaluation and recording, is further included: a weight threshold range is set according to the individual pig's situation, and the validity of the pig's weight obtained in S4 is evaluated; when the obtained weight is within the weight threshold range, the weight information is recorded; when the obtained weight is outside the weight threshold range, the weight information is discarded and the pig's weight is measured again starting from S1. Specifically, the weight threshold range can be set according to the pig's age in months, so as to exclude obviously unreasonable and invalid weight data.

[0042] like Figure 4 As shown, the present invention also provides a multimodal pig weight measurement system based on attention fusion, which is used to implement the multimodal pig weight measurement method based on attention fusion mentioned above. The multimodal pig weight measurement system based on attention fusion includes: Depth camera, used to obtain RGB images and depth images of pigs; Point cloud data conversion module, used to convert depth images into point cloud data; 2D feature extraction module, used to extract 2D features from RGB images; 3D feature extraction module, used to extract 3D features from point cloud data; The cross-modal attention fusion module is used to fuse the extracted 2D features and 3D features based on the attention mechanism to obtain attention fusion features; The MLP weight regression prediction model is used to predict the weight information of pigs based on the attention fusion features; The evaluation and recording module is used to set a weight threshold range according to the individual conditions of the pigs and to evaluate the effectiveness of the pig weight obtained by the MLP weight regression prediction model; when the obtained weight is within the weight threshold range, the weight information is recorded; when the obtained weight is outside the weight threshold range, the weight information is discarded.

[0043] The functional modules in the multimodal pig weight measurement system based on attention fusion correspond one-to-one to the steps in the above-described method; therefore, the specific information processing process will not be described here in detail.

[0044] Compared with the prior art, the present invention has the following advantages: 1. Improved measurement accuracy and robustness: The present invention simultaneously utilizes RGB images containing texture and color information and depth images containing three-dimensional spatial information, overcoming the limitations of a single modality in complex scenarios. When the pig is partially occluded or the posture is changeable, the weight prediction error of the present invention is reduced by 30%-40% compared with the traditional RGB method, which is significantly better than technologies that rely solely on monocular vision or manual measurement. The present invention also adaptively allocates the weights of RGB and depth features through an attention mechanism (such as focusing on extracting depth information in the head / torso area), avoiding feature redundancy caused by simple splicing. The sensitivity of the fused features to key parts (such as back fat thickness) is increased by 25%, which is more in line with the growth patterns of live pigs.

[0045] 2. Achieves fully automated and efficient management: The present invention combines a depth camera with an algorithm to complete pig weight measurement without human intervention, and can replace traditional contact weighing (such as a floor scale) to reduce pig herd stress reactions. The present invention can shorten the time required for a single measurement from 5 minutes per head to seconds, improving efficiency by 90%, and supports all-weather continuous monitoring. By applying the present invention to a pig breeding management system, it can automatically record weight change trends, combine behavioral analysis models (such as food intake and activity level), and generate precise feeding plans; the feed conversion rate is optimized by 10%-15%, effectively reducing breeding costs, while reducing pollution caused by overfeeding.

[0046] 3. Enhanced adaptability to complex scenarios: This invention leverages depth information to effectively distinguish overlapping targets (such as crowded pig farming scenarios) and suppresses background noise (such as manure and equipment interference) through an attention mechanism. In dense farming environments, the target segmentation accuracy of this invention is 20% higher than traditional methods, and the missed detection rate is reduced to below 5%. The MLP weight regression prediction model uses end-to-end learning to automatically adapt to the body shape characteristics of pigs of different ages and breeds without the need for manual calibration. The full-cycle measurement error, from piglets (15kg) to marketed pigs (120kg), is stable within ±3%.

[0047] 4. Improved Data Utilization: This method uses a depth camera to simultaneously capture RGB and depth images. By combining RGB images with depth point cloud data, it fully utilizes information from both modalities, improving data utilization. Furthermore, through deep learning modeling and feature extraction and fusion, it fully utilizes key information in the data, thereby improving the accuracy of weight prediction.

[0048] In summary, this invention employs a non-contact weight measurement method, completely avoiding the stress response caused by traditional contact measurement on animals, and can better meet the needs of efficient and automated monitoring in large-scale farms. By fusing 2D and 3D features to form a fusion feature with joint spatial-texture representation capabilities, this invention enhances the system's adaptability in complex scenarios and provides highly robust and discriminative input data for the subsequent MLP weight regression prediction model, significantly improving the accuracy of weight prediction.

[0049] It can be seen that the present invention has the excellent characteristics of high precision, strong robustness and explainability, and is widely applicable to scenarios such as smart farming, on-site transactions, and health monitoring. It provides a solid data foundation and decision-making support for pig weight estimation and subsequent volume and health assessment, and can effectively assist the intelligent upgrade of the farming industry.

[0050] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A multimodal pig weight measurement method based on attention fusion, characterized in that: The following steps are involved: S1. Data acquisition: Use a depth camera to synchronously acquire RGB images and depth images of pigs, and convert the depth images into point cloud data; S2, multimodal feature extraction: extract 2D features from RGB images and 3D features from point cloud data; S3, feature fusion: The extracted 2D features and 3D features are fused across modalities based on the attention mechanism to obtain attention fusion features; S4, MLP regression prediction: Input the attention fusion features into the MLP weight regression prediction model, and predict the weight of the pigs through the MLP weight regression prediction model; The MLP weight regression prediction model is obtained by the following method: S401, using the attention fusion feature as input and the pig's weight as output, establish an MLP weight regression prediction model; S402. Take several pigs as samples and generate a piece of sample data (A, W) for each sample, where A is the attention fusion feature obtained by the methods of S1 to S3, and W is the actual weight of the pig; S403: Using sample data to perform supervised training on the MLP weight regression prediction model.

2. The multimodal pig weight measurement method based on attention fusion according to claim 1 is characterized in that: Said S1 also includes, after converting the depth image into point cloud data, performing bicubic interpolation enhancement processing on the point cloud data to insert new points into the sparse point cloud and improve the point cloud density.

3. The multimodal pig weight measurement method based on attention fusion according to claim 1 is characterized in that: In S2, an improved convolutional neural network based on the ResNet50 network is used to extract 2D features from RGB images; The improved convolutional neural network is improved on the basis of the ResNet50 network. The improvements include: adding a residual attention module after the Conv1 layer of the original ResNet50 network; replacing the 3×3 standard convolution of the original ResNet50 network with a depthwise separable convolution; and introducing a spatial pyramid pooling layer after the last convolution layer of the original ResNet50 network to output an RGB global feature vector representing 2D features.

4. The multimodal pig weight measurement method based on attention fusion according to claim 3 is characterized in that: In S2, the PointNet++ network is used to extract 3D features from point cloud data.

5. The multimodal pig weight measurement method based on attention fusion according to claim 4 is characterized in that: In S3, the method of cross-modal feature fusion of the extracted 2D features and 3D features based on the attention mechanism is as follows: 2D features and 3D features are spliced ​​along the channel dimension to generate a joint feature vector; the channel importance of the spliced ​​features is recalibrated through the SE module, and the channel weights are generated after compressing the global information; the weighted features are input into the cross-attention layer, with 2D features as queries and 3D features as key-values, the cross-modal correlation is calculated, a fusion weight map is generated, and finally the attention fusion features are output.

6. The multimodal pig weight measurement method based on attention fusion according to claim 1 is characterized in that: It also includes S5, evaluation record: setting a weight threshold range according to the individual situation of the pig, and evaluating the effectiveness of the pig's weight obtained in S4; when the obtained weight is within the weight threshold range, recording the weight information; when the obtained weight is outside the weight threshold range, discarding the weight information and re-measuring the pig's weight starting from S1.

7. A multimodal pig weight measurement system based on attention fusion, used to implement the multimodal pig weight measurement method based on attention fusion according to any one of claims 1 to 6, characterized in that: include: Depth camera, used to obtain RGB images and depth images of pigs; Point cloud data conversion module, used to convert depth images into point cloud data; 2D feature extraction module, used to extract 2D features from RGB images; 3D feature extraction module, used to extract 3D features from point cloud data; The cross-modal attention fusion module is used to fuse the extracted 2D features and 3D features based on the attention mechanism to obtain attention fusion features; The MLP weight regression prediction model is used to predict the weight information of pigs based on attention fusion features.

8. The multimodal pig weight measurement system based on attention fusion according to claim 7 is characterized in that: It also includes an evaluation and recording module, which is used to set a weight threshold range according to the individual situation of the pigs and evaluate the effectiveness of the pig weight obtained by the MLP weight regression prediction model; when the obtained weight is within the weight threshold range, the weight information is recorded; when the obtained weight is outside the weight threshold range, the weight information is discarded.

Images