Intelligent meat duck weight estimation method combining deep learning and multi-modal data

Abstract

The invention discloses an intelligent meat duck weight estimation method combining deep learning and multi-modal data, and the method comprises the steps: S1, obtaining the visual modal data of a meat duck based on a multi-view camera, S2, collecting the non-visual modal data of the meat duck, and S3, fusing the visual modal data and the non-visual modal data based on a deep learning network, s4, performing meat duck key point detection based on the fusion data, and correcting the posture and shielding error; S5, estimating the weight of the meat duck by using the corrected key point data; according to the intelligent meat duck weight estimation method combining deep learning and multi-modal data, real-time weight estimation of mass meat ducks is realized, the manual operation time is saved, the production management efficiency is remarkably improved, different postures of the meat ducks and complex breeding scenes in which the meat ducks are mutually shielded can be dealt with, and the meat duck weight estimation efficiency is improved. The adaptability and robustness of the weight estimation system in a real breeding environment are enhanced, and the stability performance under various conditions is ensured.

Description

Technical Field

[0001] The present invention relates to the field of agricultural artificial intelligence technology, and in particular to an intelligent meat duck weight estimation method combining deep learning and multimodal data. Background Art

[0002] In traditional poultry farming management, the estimation of duck weight is an important link, which directly affects production efficiency and economic benefits. Existing methods for estimating duck weight mainly rely on physical measurement methods, such as weighing ducks one by one through weighing equipment. However, this method has many limitations, such as complex operation, time-consuming and labor-intensive, and difficult to achieve efficient management in large-scale farming environments. In order to solve this problem, intelligent weight estimation methods have gradually become a research hotspot, among which the duck weight estimation method combining deep learning and multimodal data has attracted much attention. This type of method uses deep learning algorithms to estimate weight by fusing images, videos, and sensor data, effectively reducing dependence on physical equipment and significantly improving weight estimation efficiency and accuracy. In particular, the application of deep learning technology enables the system to automatically extract the appearance features of ducks through training models, and infer their weight based on these features, reducing the need for human intervention.

[0003] However, the current weight estimation method based on visual modality faces some technical difficulties in practical application. In actual breeding scenarios, meat ducks are often in different postures, and due to group breeding, there is often mutual occlusion between meat ducks. These factors lead to inaccurate detection of key points of meat ducks in images or videos, which in turn affects the accuracy of weight estimation. Posture changes and occlusions will cause the deep learning model to lose some key information during feature extraction and fail to fully capture the body characteristics of meat ducks, thus causing weight estimation errors. How to improve the key point detection capability of visual modality in complex environments is still an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent meat duck weight estimation method that combines deep learning and multimodal data to solve the problem that meat ducks in the prior art are often in different postures in actual breeding scenarios, and due to group breeding, there is often mutual occlusion between meat ducks. These factors lead to inaccurate detection of key points of meat ducks in images or videos, which in turn affects the accuracy of weight estimation. Posture changes and occlusions will cause the deep learning model to lose some key information during feature extraction, and it is impossible to fully capture the body characteristics of meat ducks, thereby causing weight estimation errors.

[0005] To achieve the above object, the present invention provides the following technical solution: an intelligent meat duck weight estimation method combining deep learning and multimodal data, the method comprising:

[0006] S1. Obtain visual modality data of meat ducks based on a multi-view camera;

[0007] S2, collecting non-visual modality data of meat ducks;

[0008] S3, Fusion of visual modality data and non-visual modality data based on deep learning network;

[0009] S4, detecting key points of the duck based on the fused data and correcting posture and occlusion errors;

[0010] S5. estimating the weight of the duck using the corrected key point data;

[0011] In the step S3, the visual modality data is processed by a convolutional neural network, and the non-visual modality data is processed by a multi-layer perceptron. The feature vectors of the visual modality and the non-visual modality are concatenated, and the merged features are further learned through a fully connected layer. A self-attention mechanism is introduced to dynamically adjust the weight of each modality feature, and the multi-modal feature space is mapped to a high-dimensional space through tensor operations to capture the complex interaction characteristics between modalities.

[0012] The step S4 includes analyzing the visual modality data based on the posture recognition model, extracting and locating the key points of the skeleton of the duck, and calculating the occlusion rate R according to the position of the key points of the skeleton, where the occlusion rate R=N o / N t , where N o Indicates the number of occluded key points, N t Indicates the total number of key points expected to be detected, and determines whether the occlusion rate R exceeds the set threshold T r , when R exceeds the threshold T r , then the parameters of the detection algorithm are adjusted according to the posture in the previously stored posture database to compensate for the error;

[0013] The step S5 includes obtaining the key point coordinates of the meat duck through a deep learning network of the visual modality, reconstructing the three-dimensional skeleton structure of the meat duck, extracting body features from the three-dimensional reconstructed skeleton model, combining the body features with non-visual modality data, and estimating the weight of the meat duck through a fusion model. The weight estimation model is based on a regression tree ensemble algorithm, and uses the characteristics of multimodal data for regression prediction.

[0014] Preferably, step S1 includes installing cameras at different positions and angles to obtain multi-view images of the duck, and synchronously collecting image data from each camera to ensure that subsequent data fusion is not affected by time differences. The collected visual data is standardized and stored in a format suitable for deep learning processing, and numbered according to timestamps or collection order.

[0015] Preferably, the step S2 comprises:

[0016] The weight sensor is used to collect data on the weight of the duck in real time when the duck briefly touches the ground or the conveyor belt;

[0017] Use accelerometers or gyroscopes to collect movement behavior data of meat ducks;

[0018] Use humidity sensors to monitor the temperature and humidity of the environment where the ducks are located;

[0019] The non-visual modality data is timestamped by removing noise signals from the non-visual modality data through a low-pass filter and synchronizing the non-visual modality data with the visual data;

[0020] Store non-visual data in a structured data format.

[0021] Preferably, the self-attention mechanism calculates the importance of each modal feature according to the relationship between each element in the feature vector and other elements, and performs weighted processing on the features of each modality through the attention matrix. The specific formula is:

[0022] F att =A·F concat ,

[0023] Among them, F att represents the feature vector weighted by the self-attention mechanism, A represents the self-attention weight matrix, and F conat Represents the concatenated feature vector.

[0024] Preferably, the attention-weighted feature F att By mapping to high-dimensional space through tensor operations, the complex interaction characteristics between visual and non-visual modalities are captured. The tensor mapping formula is:

[0025] F tensor =T(F att )=W·F att ,

[0026] Among them, F tensor Represents the high-dimensional feature vector after tensor operation, T(F att ) represents the tensor mapping function, W represents the weight matrix, F att Represents the feature vector weighted by the self-attention mechanism.

[0027] Preferably, the analyzing of visual modality data according to the posture recognition model further includes:

[0028] Delineate the region of interest (ROI) containing the duck to be detected in the image;

[0029] Use a pre-trained neural network model to perform refined detection on key points in the ROI;

[0030] updating the mask region according to the detected key point positions, and treating the key points outside the mask region as occluded points;

[0031] The data provided by non-visual sensors is used to compensate for occlusion. o >k×N t The auxiliary data set verification and correction mechanism is started when N o Indicates the number of occluded key points, k represents a coefficient less than 1, N t Indicates the total number of key points expected to be detected.

[0032] Preferably, the auxiliary correction of key point errors based on non-visual data includes:

[0033] Inputting non-visual sensor information into the auxiliary pose estimator model to predict potential missing or misdetected parts;

[0034] Reconstruct the bone structure diagram by combining the predicted results and visual data;

[0035] Estimate the pose angle A again based on the redefined key point set est , the estimation uses an optimized loss function formula to ensure that relatively accurate values ​​are obtained even in the presence of occlusion;

[0036] Determine the posture deviation E, that is, the estimated A est Compared with the known good sample set standard pose A ref Is the gap between them smaller than a certain tolerance E? m If yes, then continue to enter the weight estimation circuit to estimate the weight.

[0037] Preferably, when the key points need to be corrected by re-estimating the posture, the steps include:

[0038] Filter out multiple sensor signals X that affect posture n =(a,b,c...) to optimize the prediction;

[0039] Use a fusion algorithm that recalculates key points based on an adaptive learning rate adjustment strategy;

[0040] According to the formula D = f(N o ,T) measures the amount of keypoint adjustment D, where T represents the pre-training time step, in which case it is interpreted as a confidence indicator or stability coefficient of the model;

[0041] If the key point change amplitude D is lower than a certain standard ΔT thresh , it is considered that the posture correction reaches the expected state, otherwise the process is repeated and the correction is continued until it is stable.

[0042] Preferably, when the optimized fusion model is used to adjust the detection deviation caused by insufficient visual information, it also includes:

[0043] Collect a reference sample dataset L of the sizes of various parts of the duck and compare it with the size dataset I of the current instance;

[0044] Based on the degree of set matching P m =1 / (1+exp(s×(SL)) / k 1 )) to quantify the quality factor of the match;

[0045] where s and k 1 are adjustment constants, S is the size vector of the sample, and L is the actual size data point measured for this duck;

[0046] The comparison factor is higher than the preset trust level P thres , that is, P m >Pt hres Indicates high confidence;

[0047] If not satisfied, return to obtain data from additional viewing angles or adjust existing model parameters and try again.

[0048] Preferably, the characteristics of the multimodal data are used to predict the weight of the meat duck, and the input of the model includes body shape features extracted from visual data and non-visual modal data. The regression tree integration formula is specifically:

[0049]

[0050] in, represents the predicted weight of the meat duck, X represents the input feature vector, and f i (X) represents the prediction result of the i-th regression tree, α i represents the weight coefficient of the i-th tree, n represents the number of regression trees, and i represents the i-th regression tree.

[0051] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0052] This intelligent meat duck weight estimation method that combines deep learning and multimodal data obtains visual modal data of meat ducks based on a multi-view camera, collects non-visual modal data of meat ducks, fuses visual modal data with non-visual modal data based on a deep learning network, detects key points of meat ducks based on the fused data, corrects posture and occlusion errors, and estimates the weight of meat ducks using the corrected key point data, reducing errors caused by posture changes and occlusions, thereby greatly improving the accuracy of weight estimation. It does not need to rely on traditional weighing equipment, reduces cumbersome physical measurement steps, realizes non-contact and non-destructive meat duck weight estimation, reduces operating difficulty and improves weight estimation efficiency. It is suitable for large-scale farms and can quickly process large amounts of data to achieve Real-time weight estimation of large quantities of meat ducks saves manual operation time and significantly improves production management efficiency. It can cope with complex breeding scenarios with different postures and mutual occlusion of meat ducks, enhances the adaptability and robustness of the weight estimation system in real breeding environments, and ensures stable performance under various conditions. It solves the problem that meat ducks in the existing technology are often in different postures in actual breeding scenarios, and due to group breeding, there is often mutual occlusion between meat ducks. These factors lead to inaccurate detection of key points of meat ducks in images or videos, which in turn affects the accuracy of weight estimation. Posture changes and occlusions will cause the deep learning model to lose some key information during feature extraction, and cannot fully capture the body characteristics of meat ducks, thereby causing weight estimation errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, an intelligent meat duck weight estimation method combining deep learning and multimodal data includes the following steps: S1, obtaining visual modal data of meat ducks based on a multi-view camera; S2, collecting non-visual modal data of meat ducks; S3, fusing visual modal data with non-visual modal data based on a deep learning network; S4, detecting key points of meat ducks based on the fused data and correcting posture and occlusion errors; S5, estimating the weight of meat ducks using the corrected key point data.

[0056] In step S1, a multi-view camera is used to shoot ducks to capture visual modal data from different angles. In step S2, non-visual modal data is obtained through sensors, including environmental factors such as weight, acceleration, temperature and humidity. In step S3, the visual modal data is processed by a convolutional neural network, and the non-visual modal data is processed by a multi-layer perceptron. Finally, the feature vectors of the two modalities are concatenated and further learned through a fully connected layer. In order to improve the feature fusion effect, a self-attention mechanism is introduced to dynamically optimize the modal features through weight adjustment, and the fused multi-modal features are mapped to a high-dimensional space using tensor operations to capture the complex interaction characteristics between modalities.

[0057] In step S4, the posture recognition model is used to analyze the visual modal data, extract the key points of the duck skeleton, calculate the occlusion rate R to detect the occlusion situation, and if the occlusion rate exceeds the preset threshold Tr, the stored posture database is used to compensate for the error. In step S5, based on the corrected key point data and combined with the non-visual modal information, the three-dimensional skeleton structure of the duck is reconstructed, and the weight of the duck is estimated by the regression tree integration algorithm. The present invention can effectively improve the accuracy of the duck weight estimation through the multimodal data fusion of the deep learning network. The self-attention mechanism is used to dynamically adjust the modal features to ensure that the complex relationship between different modal features is optimized, thereby improving the generalization ability of the weight estimation model. In addition, the steps of correcting the posture and occlusion errors ensure that the weight of the duck can be accurately detected in different postures and environments. At the same time, the introduction of the regression tree integration algorithm further improves the prediction accuracy of the weight estimation model.

[0058] In the above scheme, the processing of visual modality data can be replaced by other suitable neural network architectures, such as ResNet or Transformer models. The processing of non-visual modality data can be obtained through other sensor technologies, such as lidar or ultrasonic sensors. For the self-attention mechanism, the dynamic adjustment capability of feature weights can be further enhanced by introducing a multi-head attention mechanism. In addition, the regression tree ensemble algorithm can also be replaced by other regression algorithms, such as gradient boosted decision tree (GBDT) or random forest, to achieve similar effects.

[0059] In a possible implementation, step S1 includes installing multiple cameras at different positions and angles to obtain multi-view images of the duck. The image data of each camera is collected synchronously to ensure that the subsequent data fusion is not affected by time differences. The collected visual data is standardized and stored in a format suitable for deep learning processing. The data is numbered according to the timestamp or acquisition order. Multi-view images of the duck are obtained by cameras installed at different angles to ensure that complete visual information can be captured at various angles. In order to avoid data deviations caused by different acquisition times of different cameras, the data of each camera is collected synchronously. The visual modality data is standardized to ensure the consistency of the image data, and is organized in chronological order through a numbering system.

[0060] Through the synchronous acquisition of multi-view cameras, visual data of meat ducks from different angles can be obtained to avoid information loss caused by a single perspective. At the same time, through data standardization, consistency and comparability are ensured when data is input into the deep learning network, thereby improving the subsequent weight estimation accuracy.

[0061] The camera installation method can be adjusted according to the needs of the environment, for example, a mobile camera system can be used to further increase flexibility. The data can also be standardized according to the processing requirements, using other processing techniques such as normalization or contrast enhancement.

[0062] In a possible implementation, step S2 includes collecting weight data of the meat duck when it briefly contacts the ground or conveyor belt in real time through a weight sensor; collecting movement behavior data of the meat duck using an accelerometer or a gyroscope; monitoring the temperature and humidity of the environment through a humidity sensor; and removing noise signals of non-visual modality data through a low-pass filter, synchronizing it with visual modality data, and storing it in a structured data format. During the non-visual modality data collection process, the weight sensor obtains the weight information of the meat duck in real time, and the sensor collects weight data when the meat duck briefly contacts the ground or conveyor belt. The accelerometer and gyroscope are used to monitor the movement behavior of the meat duck, while the humidity sensor monitors the environmental conditions. To ensure the accuracy of the data, a low-pass filter is used to filter the non-visual modality data to remove noise signals, and the data is synchronized with the visual modality data through a timestamp. The combination of multiple non-visual sensor data can further enhance the accuracy of the meat duck weight estimation. At the same time, the use of a low-pass filter effectively removes noise signals and ensures the authenticity and validity of the data. In addition, the synchronous processing of non-visual data and visual data ensures the time consistency during data fusion.

[0063] The acquisition method of non-visual modality data can be adjusted according to the actual application. For example, the weight sensor can be replaced by a laser rangefinder or other measuring equipment. For the data synchronization process, the accuracy of data processing can be further improved by various filters (such as Kalman filter).

[0064] In one possible implementation, the visual modality data in step S3 is processed by a convolutional neural network (CNN), and the non-visual modality data is processed by a multi-layer perceptron (MLP). The visual modality data extracts spatial features from the image through CNN, and generates a high-level feature map through multi-layer convolution and pooling operations. Non-visual modality data, such as weight, acceleration, humidity, etc., are processed by MLP, which consists of multiple layers of fully connected layers, each of which is followed by a nonlinear activation function to extract relevant features from these structured data. Subsequently, the feature vectors of the visual modality and the non-visual modality are concatenated, and the merged features are further learned through the fully connected layer.

[0065] In this process, the self-attention mechanism is introduced to dynamically adjust the weight of each modal feature. Specifically, the self-attention mechanism generates a weight matrix based on the relationship between each element in the concatenated feature vector and other elements, which is used to weight the features of different modalities. The attention weight can identify which modal features are more important to the task goal (such as weight estimation), thereby strengthening these important features and weakening irrelevant or redundant information.

[0066] The weighted multimodal features are mapped to a high-dimensional space through tensor operations. Tensor operations are used to model the complex interactions between different modal features and can capture the underlying complex relationships between modalities. The tensor mapping function maps the weighted feature vectors to a higher-dimensional space to enhance the model's ability to express modal interactions. In this way, the model can make full use of visual and non-visual modal data and capture the complex interactive characteristics between them, thereby improving the accuracy of duck weight estimation.

[0067] The convolutional neural network extracts visual features through multi-layer convolution, and the MLP processes non-visual data. The features of the two are concatenated and enter the fully connected layer for joint learning. The self-attention mechanism dynamically adjusts the feature weights of different modalities by calculating the relationship between feature vectors, and enhances the ability to capture modal interactions through tensor mapping. This implementation method effectively integrates visual and non-visual modal data, and optimizes the distribution of feature weights through the self-attention mechanism to ensure that the effective information of each modality is fully utilized. Tensor operations further strengthen the modeling of complex interactions between modalities, significantly improving the accuracy of duck weight estimation.

[0068] In this embodiment, the convolutional neural network can be replaced by other visual processing network architectures, such as ResNet or Vision Transformer, and the processing of non-visual modality data can use more complex deep learning networks, such as recurrent neural networks (RNN). In addition, the self-attention mechanism can further enhance the diversity and flexibility of feature capture through the multi-head attention mechanism.

[0069] The self-attention mechanism calculates the importance of each modal feature based on the relationship between each element in the feature vector and other elements, and weights the features of each modality through the attention matrix. The specific formula is:

[0070] F att =A·F concat ,

[0071] Among them, F att represents the feature vector weighted by the self-attention mechanism, A represents the self-attention weight matrix, and F concat Represents the concatenated feature vector.

[0072] The self-attention mechanism calculates the element relationship in each modal feature vector to obtain the relative weight of each modal feature. The feature weight matrix is ​​generated by comparing the similarity and difference of each modal feature. Subsequently, the weight matrix is ​​used to weight the spliced ​​multimodal features. The result is a weighted feature vector that can better capture the complex interactive relationship between visual modality and non-visual modality data. By introducing the self-attention mechanism, this method can dynamically adjust the weight of multimodal features so that important features are enhanced and irrelevant features are weakened. This mechanism not only improves the model's ability to understand the complex interactive relationship between features, but also effectively improves the accuracy of the weight estimation results.

[0073] In this scheme, the calculation method of the self-attention mechanism can be replaced by a multi-head attention mechanism to simultaneously capture feature relationships in more dimensions. The concatenation method of feature vectors can also be adjusted according to the specific network architecture, for example, feature weighted summation or convolution-based fusion can be used.

[0074] The feature F after attention weighting att , mapped to high-dimensional space through tensor operations, capturing the complex interaction characteristics between visual and non-visual modalities. The tensor mapping formula is:

[0075] F tensor =T(F att )=W·F att ,

[0076] Among them, F tensorRepresents the high-dimensional feature vector after tensor operation, T(F att ) represents the tensor mapping function, W represents the weight matrix, F att Represents the feature vector weighted by the self-attention mechanism.

[0077] By performing tensor operations on the weighted feature vectors, they can be mapped to a higher-dimensional feature space to further capture the complex interactions between multimodal data. The tensor mapping function is used to combine multimodal features together, and the weight matrix is ​​used to adjust the mapping effect of each modal feature in the high-dimensional space, so that the final generated high-dimensional features contain richer information. By mapping the features to the high-dimensional space through tensor operations, the complex associations between visual and non-visual modalities can be better captured, the model's ability to express multimodal data can be improved, and the prediction effect on the weight of meat ducks can be enhanced. In addition, the use of high-dimensional features can also improve the robustness and generalization ability of the model.

[0078] The mapping function of tensor operations can be implemented in different ways, such as linear mapping, nonlinear activation function, etc. The dimension of the mapped high-dimensional space can also be adjusted according to the amount of data and computing resources. In addition, the weight matrix can be optimized through automatic learning strategies, such as continuously adjusting the weight value through the gradient descent algorithm.

[0079] In a possible implementation, step S4 includes analyzing the visual modality data based on a posture recognition model to extract and locate the key points of the bones of the duck. The posture recognition model uses a deep learning method to identify the bone structure of the duck from the visual image and locate multiple key points, such as the joint positions of the head, wings, legs, and tail.

[0080] According to the positions of these key points, the occlusion rate R is calculated. The formula for the occlusion rate is occlusion rate R = N o / N t , where N o Indicates the number of occluded key points, N t Indicates the total number of key points that are expected to be detected. When the corresponding bone key points of the duck are not correctly detected due to occlusion or other interference in some parts of the image, the system will calculate the number of occluded key points N. o , and calculate the occlusion rate R accordingly.

[0081] Next, determine whether the occlusion rate R exceeds the set threshold T r If the occlusion rate R exceeds the predetermined threshold T r, the system will dynamically adjust the parameters of the detection algorithm based on the historical data of the relevant posture in the posture database previously stored to compensate for the errors caused by occlusion. The posture database contains typical posture information of meat ducks in different postures, angles and viewing angles. By comparing the key point data currently detected with the known postures in the posture database, the algorithm can adjust the parameters of the detection model to correct the key point positioning errors caused by occlusion.

[0082] The visual modality data is processed by the posture recognition model, and the model extracts the key points of the duck bones. The system calculates the occlusion rate R in real time to determine whether the key points are occluded and the severity of the occlusion. If the occlusion rate RRR exceeds the threshold T r The system will adjust the parameters of the detection algorithm based on the pre-stored posture database to compensate for posture recognition errors and ensure the accuracy of key point detection. This implementation method can effectively deal with the occlusion problem that may occur in meat ducks in natural environments. By calculating the occlusion rate, the system can automatically identify when detection compensation is needed to ensure that the posture of the meat duck can still be accurately located even when some key points are blocked. Dynamic adjustment through historical data of the posture database improves the stability and robustness of posture recognition, thereby enhancing the accuracy of weight estimation.

[0083] In this embodiment, the calculation method of the occlusion rate can be optimized by different methods, such as using a weighted occlusion rate to give a higher weight to more important key points. The posture recognition model can choose other deep learning frameworks, such as a posture recognition algorithm based on a graph convolutional network (GCN) to enhance the detection capability of complex postures. In addition, the compensation algorithm of the posture database can be dynamically adjusted by a Bayesian reasoning-based method to further improve the posture detection accuracy under occlusion conditions.

[0084] Analyzing visual modality data based on the gesture recognition model also includes:

[0085] Delineate the region of interest (ROI) containing the duck to be detected in the image;

[0086] Use a pre-trained neural network model to perform refined detection on key points in the ROI;

[0087] updating the mask region according to the detected key point positions, and treating the key points outside the mask region as occluded points;

[0088] The data provided by non-visual sensors is used to compensate for occlusion. o >k×N t The auxiliary data set verification and correction mechanism is started when N o Indicates the number of occluded key points, k represents a coefficient less than 1, N tIndicates the total number of key points expected to be detected.

[0089] First, the region of interest (ROI) is defined in the image to ensure that only the ducks in this region are detected. Then, the key points in the region of interest are finely detected using a pre-trained neural network. The mask area is used to mark the detected occluded parts, and the key points outside the occluded points are used as valid data. The occlusion rate R of the key points is detected to determine whether the correction mechanism needs to be activated. When the occluded points exceed the set ratio k×N t When the non-visual modality sensor data is used to compensate for the detection results, the accuracy of posture detection can be maintained even in the case of severe occlusion. This method can accurately detect the posture of the duck in the case of occlusion by combining visual and non-visual data. The compensation mechanism of non-visual data ensures the robustness of the detection and improves the stability and accuracy of the weight estimation process.

[0090] The ROI delineation method can be optimized through automated algorithms, such as dynamically adjusting the size of the region of interest to adapt to different scenarios. The detection mechanism of the mask region can also be enhanced through improved occlusion processing algorithms, such as density estimation methods. In addition, the compensation algorithm can use Bayesian network-based reasoning methods to further enhance its prediction effect.

[0091] The auxiliary correction of key point errors based on non-visual data includes:

[0092] Inputting non-visual sensor information into the auxiliary pose estimator model to predict potential missing or misdetected parts;

[0093] Reconstruct the bone structure diagram by combining the predicted results and visual data;

[0094] Estimate the pose angle A again based on the redefined key point set est , the estimation uses an optimized loss function formula to ensure that relatively accurate values ​​are obtained even in the presence of occlusion;

[0095] Determine the posture deviation E, that is, the estimated A est Compared with the known good sample set standard pose A ref Is the gap between them smaller than a certain tolerance E? m If yes, then continue to enter the weight estimation circuit to estimate the weight.

[0096] The non-visual data input aids the pose estimator model to predict the missing or misdetected parts in the visual modality data. The skeletal structure of the duck is reconstructed based on the prediction results and the visual data. Subsequently, the pose angle A of the duck is estimated based on the newly defined keypoint set. est, the posture angle is corrected through the optimized loss function to ensure that the posture is still accurate under occlusion. Finally, the posture deviation E is compared to see if it is within the tolerance range. If it meets the conditions, the weight estimation process continues. By introducing non-visual data for auxiliary correction, the posture estimation accuracy under occlusion can be effectively improved. The optimized loss function further ensures the robustness and stability of posture estimation in different scenarios, thereby providing more accurate basic data for subsequent weight estimation.

[0097] The auxiliary pose estimator model can adopt other machine learning algorithms, such as random forest or extreme gradient boosting tree, to further improve the accuracy of prediction. The loss function of pose estimation can introduce weighted loss to prioritize more important pose parameters.

[0098] When it is necessary to correct key points by re-estimating the posture, it includes:

[0099] Filter out multiple sensor signals X that affect posture n =(a,b,c...) to optimize the prediction;

[0100] Use a fusion algorithm that recalculates key points based on an adaptive learning rate adjustment strategy;

[0101] According to the formula D = f(N o ,T) measures the amount of keypoint adjustment D, where T represents the pre-training time step, in which case it is interpreted as a confidence indicator or stability coefficient of the model;

[0102] If the key point change amplitude D is lower than a certain standard ΔT thresh , it is considered that the posture correction reaches the expected state, otherwise the process is repeated and the correction is continued until it is stable.

[0103] In the posture correction process, the data X collected by the sensor is first n The key point adjustment amount D is given by the formula D = f(N o ,T), where T in the formula is used as the confidence or stability coefficient of the model to determine the reliability of the correction result. If the change amplitude D of the key point is lower than the set threshold ΔT thresh, indicating that the posture correction has achieved the expected accuracy, otherwise it continues to iterate and correct. By adjusting the adaptive learning rate, this method can dynamically adjust the learning rate of the model during the correction process, thereby improving the efficiency and accuracy of posture correction. The fusion of sensor data and deep learning algorithms enhances the correction capability for complex postures, ensuring that the system can still accurately estimate the weight of the duck even after multiple adjustments. In addition, this method further ensures the reliability of the correction results by introducing a confidence index during the correction process.

[0104] The adaptive learning rate can use other optimization algorithms, such as the Adam optimizer, to improve the stability of posture correction. The selection of sensor signals can also be adjusted based on different environmental conditions, such as introducing more sensor data (such as 3D scanners, etc.) in more complex environments to enhance the adaptability of the model. In addition, the judgment criterion for posture correction can be dynamically adjusted by adjusting the threshold ΔT thresh To further optimize.

[0105] When using the optimized fusion model to adjust the detection bias caused by insufficient visual information, it also includes:

[0106] Collect a reference sample dataset L of the sizes of various parts of the duck and compare it with the size dataset I of the current instance;

[0107] Based on the degree of set matching P m =1 / (1+exp(s×(SL)) / k 1 )) to quantify the quality factor of the match;

[0108] where s and k 1 are adjustment constants, S is the size vector of the sample, and L is the actual size data point measured for this duck;

[0109] The comparison factor is higher than the preset trust level P thres , that is, P m >Pt hres Indicates high confidence;

[0110] If not satisfied, return to obtain data from additional viewing angles or adjust existing model parameters and try again.

[0111] In the process of weight estimation, we collect and establish a reference sample dataset L of the dimensions of different parts of the duck, and then compare the currently detected dimension data I with the sample dataset. The quality factor P of the comparison match m The degree of set matching P is based on the formula m =1 / (1+exp(s×(SL)) / k 1 )) to quantify the quality factor calculation of the match, where s and k 1are adjustment constants, S is the sample size vector, L is the actual size data point measured for this duck. If the match is higher than the set confidence level P thres , indicating that the current detection result is reliable, otherwise it will return and perform additional shooting or adjust the model parameters to obtain higher quality data. Through comparative analysis of the sizes of various parts of meat ducks, this method can effectively reduce the detection bias caused by incomplete or blurred visual data and improve the accuracy of weight estimation. Using quality factors to judge the credibility of the detection results helps to dynamically adjust the shooting and data processing strategies to ensure that the final weight estimation results are reliable. In addition, this method can dynamically adjust the model according to the matching results, further improving the flexibility and adaptability of the system.

[0112] The reference sample dataset LLL can be expanded by collecting data in more dimensions, for example, by combining the volume or body shape characteristics of meat ducks. The calculation formula of the matching quality factor can adopt other distance metrics, such as cosine similarity or Euclidean distance, to adapt to different data structures. In addition, for instances that do not meet the matching requirements, the shooting effect can be improved by introducing enhanced image processing techniques (such as high-resolution imaging).

[0113] The heavy model is based on the regression tree integration algorithm and uses the characteristics of multimodal data to predict the weight of meat ducks. The input of the model includes body shape features extracted from visual data and non-visual modal data. The regression tree integration formula is as follows:

[0114]

[0115] in, represents the predicted weight of the meat duck, X represents the input feature vector, and f i (X) represents the prediction result of the i-th regression tree, α i represents the weight coefficient of the i-th tree, n represents the number of regression trees, and i represents the i-th regression tree.

[0116] Through the regression tree ensemble algorithm, this method can integrate the prediction results of multiple regression trees, reduce the overfitting problem that may exist in a single model, and significantly improve the accuracy of weight estimation. The use of a fusion of visual and non-visual modality data further enriches the input features and enhances the prediction ability of the model. At the same time, the weight coefficient αi\alpha_iαi in the model is dynamically adjusted through training to ensure the optimal prediction effect. The regression tree ensemble algorithm can be replaced by other ensemble learning algorithms, such as gradient boosting tree (GBDT) or random forest, to further improve the prediction accuracy. The way the feature vector is constructed can be optimized according to specific application requirements, such as adding more feature dimensions related to the body shape of meat ducks. In addition, the number of regression trees N and their weight coefficient α i Different optimization strategies can be adjusted to achieve better weight estimation results.

[0117] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent meat duck weight estimation method combining deep learning and multimodal data, characterized in that: The method comprises: S1. Obtain visual modality data of meat ducks based on a multi-view camera; S2, collecting non-visual modality data of meat ducks; S3, Fusion of visual modality data and non-visual modality data based on deep learning network; S4, detecting key points of the duck based on the fused data and correcting posture and occlusion errors; S5. estimating the weight of the duck using the corrected key point data; In the step S3, the visual modality data is processed by a convolutional neural network, and the non-visual modality data is processed by a multi-layer perceptron. The feature vectors of the visual modality and the non-visual modality are concatenated, and the merged features are further learned through a fully connected layer. A self-attention mechanism is introduced to dynamically adjust the weight of each modality feature, and the multi-modal feature space is mapped to a high-dimensional space through tensor operations to capture the complex interaction characteristics between modalities. The step S4 includes analyzing the visual modality data based on the posture recognition model, extracting and locating the key points of the skeleton of the duck, and calculating the occlusion rate R according to the position of the key points of the skeleton, where the occlusion rate R=N o / N t , where N o Indicates the number of occluded key points, N t Indicates the total number of key points expected to be detected, and determines whether the occlusion rate R exceeds the set threshold T r , when R exceeds the threshold T r , then the parameters of the detection algorithm are adjusted according to the posture in the previously stored posture database to compensate for the error; The step S5 includes obtaining the key point coordinates of the meat duck through a deep learning network of the visual modality, reconstructing the three-dimensional skeleton structure of the meat duck, extracting body features from the three-dimensional reconstructed skeleton model, combining the body features with non-visual modality data, and estimating the weight of the meat duck through a fusion model. The weight estimation model is based on a regression tree ensemble algorithm, and uses the characteristics of multimodal data for regression prediction.

2. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 1, characterized in that: The step S1 includes installing cameras at different positions and angles to obtain multi-view images of the duck, synchronously collecting image data from each camera to ensure that subsequent data fusion is not affected by time differences, and standardizing the collected visual data, storing it in a format suitable for deep learning processing, and numbering it according to timestamps or collection order.

3. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 1, characterized in that: The step S2 comprises: The weight sensor is used to collect data on the weight of the duck in real time when the duck briefly touches the ground or the conveyor belt; Use accelerometers or gyroscopes to collect movement behavior data of meat ducks; Use humidity sensors to monitor the temperature and humidity of the environment where the ducks are located; The non-visual modality data is timestamped by removing noise signals from the non-visual modality data through a low-pass filter and synchronizing the non-visual modality data with the visual data; Store non-visual data in a structured data format.

4. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 1, characterized in that: The self-attention mechanism calculates the importance of each modal feature according to the relationship between each element in the feature vector and other elements, and performs weighted processing on the features of each modality through the attention matrix. The specific formula is: F att =A·F concat , Among them, F att , represents the feature vector weighted by the self-attention mechanism, A represents the self-attention weight matrix, F concat Represents the concatenated feature vector.

5. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 4 is characterized in that: The attention-weighted feature F att By mapping to high-dimensional space through tensor operations, the complex interaction characteristics between visual and non-visual modalities are captured. The tensor mapping formula is: F tensor =T(F att )=W·F att , Among them, F tensor Represents the high-dimensional feature vector after tensor operation, T(F att ) represents the tensor mapping function, W represents the weight matrix, F att Represents the feature vector weighted by the self-attention mechanism.

6. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 1, characterized in that: The step of analyzing the visual modality data according to the posture recognition model also includes: Delineate the region of interest (ROI) containing the duck to be detected in the image; Use a pre-trained neural network model to perform refined detection on key points in the ROI; updating the mask region according to the detected key point positions, and treating the key points outside the mask region as occluded points; The data provided by non-visual sensors is used to compensate for occlusion. o >k×N t The auxiliary data set verification and correction mechanism is started when N o Indicates the number of occluded key points, k represents a coefficient less than 1, N t Indicates the total number of key points expected to be detected.

7. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 6, characterized in that: The auxiliary correction of key point errors based on non-visual data includes: Inputting non-visual sensor information into the auxiliary pose estimator model to predict potential missing or misdetected parts; Reconstruct the bone structure diagram by combining the predicted results and visual data; Estimate the pose angle A again based on the redefined key point set est , the estimation uses an optimized loss function formula to ensure that relatively accurate values ​​are obtained even in the presence of occlusion; Determine the posture deviation E, that is, the estimated A est Compared with the known good sample set standard pose A ref Is the gap between them smaller than a certain tolerance E? m If yes, then continue to enter the weight estimation circuit to estimate the weight.

8. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 7, characterized in that: When the key points need to be corrected by re-estimating the posture, the following steps are included: Filter out multiple sensor signals X that affect posture n =(a,b,c...) to optimize the prediction; Use a fusion algorithm that recalculates key points based on an adaptive learning rate adjustment strategy; According to the formula D = f(N o ,T) measures the amount of keypoint adjustment D, where T represents the pre-training time step, in which case it is interpreted as a confidence indicator or stability coefficient of the model; If the key point change amplitude D is lower than a certain standard ΔT thresh , it is considered that the posture correction reaches the expected state, otherwise the process is repeated and the correction is continued until it is stable.

9. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 8, characterized in that: When the optimized fusion model is used to adjust the detection deviation caused by insufficient visual information, it also includes: Collect a reference sample dataset L of the sizes of various parts of the duck and compare it with the size dataset I of the current instance; Based on the degree of set matching P m =1 / (1+exp(s×(SL)) / k1)) to quantify the quality factor of the match; Where s and k1 are adjustment constants, S is the size vector of the sample, and L is the actual size data point measured for this duck; The comparison factor is higher than the preset trust level P thres , that is, P m >Pt hres Indicates high confidence; If not satisfied, return to obtain data from additional viewing angles or adjust existing model parameters and try again.

10. The intelligent meat duck weight estimation method combining deep learning and multimodal data according to claim 1, characterized in that: The weight model is based on the regression tree integration algorithm and uses the characteristics of multimodal data to predict the weight of meat ducks. The input of the model includes body shape features extracted from visual data and non-visual modal data. The regression tree integration formula is as follows: in, represents the predicted weight of the meat duck, X represents the input feature vector, and f i (X) represents the prediction result of the i-th regression tree, α i represents the weight coefficient of the i-th tree, n represents the number of regression trees, and i represents the i-th regression tree.

Images