Egg-laying hen health state monitoring method and system based on multi-modal image recognition

Abstract

The application provides a laying hen health state monitoring method and system based on multi-modal image recognition, relates to the technical field of intelligent breeding, and comprises the following steps: acquiring multi-dimensional image data of a close-up image of a crown of a laying hen, a body panoramic image and a moving shooting video; then, feature extraction is performed on the crown image to obtain blood circulation state and texture abnormality features; feather damage edges and parasite positioning are performed on the body image to generate a binary pathological feature map; abnormal behavior analysis is performed on a video sequence to obtain a space-time feature sequence. Next, the above multi-modal features are fused to construct a health state atlas. Finally, time series analysis and health scoring are performed based on the atlas, feature contribution degrees are quantified through a graph attention mechanism, and a diagnosis report containing specific disease risk labels and health scores is output. The application realizes non-contact and all-around monitoring and early warning of the health state of laying hens.

Description

Technical Field

[0001] This invention relates to the field of intelligent farming technology, and more specifically, to a method and system for monitoring the health status of laying hens based on multimodal image recognition. Background Technology

[0002] In large-scale egg-laying hen farming, timely and accurate monitoring of individual health status is crucial for preventing disease transmission, ensuring animal welfare, and improving production efficiency. Current technologies primarily rely on manual inspections and experience-based judgment, which suffers from high subjectivity, low efficiency, and the inability to achieve continuous 24-hour monitoring. Furthermore, existing monitoring methods have significant limitations: First, the health status of chickens is a complex system comprised of physiological indicators, body surface characteristics, and behavioral patterns; information from a single source cannot comprehensively and accurately reflect their health status. Second, existing technologies typically analyze various characteristics in isolation, failing to consider the intrinsic correlation between abnormalities in the comb, body surface damage, and behavioral abnormalities, leading to a high rate of misdiagnosis or missed diagnosis of potential diseases and making it difficult to form reliable comprehensive diagnostic conclusions.

[0003] Therefore, there is an urgent need for a method and system for monitoring the health status of laying hens based on multimodal image recognition to solve the above-mentioned technical problems. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for monitoring the health status of laying hens based on multimodal image recognition, so as to improve the above-mentioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows:

[0005] In a first aspect, this application provides a method for monitoring the health status of laying hens based on multimodal image recognition, including:

[0006] Acquire close-up images of the hens' combs, panoramic images of the hens' bodies, and videos of the hens moving.

[0007] Based on the close-up image of the hen's comb, feature extraction processing was performed to obtain abnormal image features and texture abnormal image features of the hen's comb blood circulation status.

[0008] Based on the panoramic image of the body, the edges of feather damage and the areas where parasites attach to the laying hen are located, resulting in a binary pathological feature map containing feather pecking marks and parasite distribution information.

[0009] Abnormal behavior feature analysis is performed based on video footage of laying hens moving around. Specifically, abnormal behavior features of laying hens are obtained by analyzing reconstruction error, resulting in a spatiotemporal feature sequence of abnormal behavior of laying hens.

[0010] Based on the abnormal image features, texture abnormal image features, binarized pathological feature map and abnormal behavior spatiotemporal feature sequence of the blood circulation status of the coronary of the laying hen, feature interaction processing is performed to construct a health status map with pathological regions as nodes and spatiotemporal correlations as edges.

[0011] Based on the health status map, time series analysis and health scoring are performed. The contribution of all features is quantified through a graph attention mechanism to obtain a diagnostic report that includes specific disease risk labels and health scores.

[0012] Secondly, this application also provides a multimodal image recognition-based system for monitoring the health status of laying hens, comprising:

[0013] The acquisition unit is used to acquire close-up images of the hen's comb, panoramic images of the hen's body, and video footage of the hen's movement.

[0014] The extraction unit is used to perform feature extraction processing based on the close-up image of the hen's crown to obtain abnormal image features and texture abnormal image features of the hen's crown blood circulation status.

[0015] The positioning unit is used to locate the edge of feather damage and the area where parasites attach to the laying hen based on the panoramic image of the body, and to obtain a binary pathological feature map containing feather pecking marks and parasite distribution information.

[0016] The analysis unit is used to perform abnormal behavior feature analysis based on the video of the movement of laying hens. Specifically, the abnormal behavior features of laying hens are analyzed by reconstruction error analysis to obtain the spatiotemporal feature sequence of abnormal behavior of laying hens.

[0017] The construction unit is used to perform feature interaction processing based on the abnormal image features, texture abnormal image features, binarized pathological feature map and abnormal behavior spatiotemporal feature sequence of the blood circulation status of the coronary of the laying hen, and to construct a health status map with pathological regions as nodes and spatiotemporal associations as edges.

[0018] The diagnostic unit is used to perform time series analysis and health scoring based on the health status map. The contribution of all features is quantified through a graph attention mechanism to obtain a diagnostic report that includes specific disease risk labels and health scores.

[0019] The beneficial effects of this invention are as follows:

[0020] This invention acquires multi-dimensional features encompassing blood circulation, body surface integrity, and behavioral dynamics by collaboratively analyzing close-up images of the comb, panoramic images of the body, and behavioral video sequences of laying hens. A graph structure is used to deeply fuse and correlate these multi-modal features, constructing a health status map with pathological regions as nodes and spatiotemporal correlations as edges, thereby characterizing the interactions between different health indicators. Furthermore, a temporal graph attention mechanism is used to dynamically quantify the contribution of each feature, generating a comprehensive diagnostic report containing specific disease risk labels and health scores. This achieves non-contact, comprehensive, and highly accurate automated monitoring and early warning of the health status of laying hens.

[0021] This invention maps multiple pathological features of laying hens into a unified graph structure space, enabling quantitative analysis of the intrinsic correlation between different health indicators. This allows for accurate analysis of the laying hen's condition, preventing misjudgments. Furthermore, by intelligently analyzing the health status of laying hens, it provides clear judgment results for general farmers.

[0022] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the process for monitoring the health status of laying hens based on multimodal image recognition, as described in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of the multimodal image recognition-based egg-laying hen health status monitoring system described in this embodiment of the invention.

[0026] In the diagram: 701, Acquisition Unit; 702, Extraction Unit; 703, Positioning Unit; 704, Analysis Unit; 705, Construction Unit; 706, Diagnosis Unit. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0028] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] Example 1:

[0030] This embodiment provides a method for monitoring the health status of laying hens based on multimodal image recognition.

[0031] See Figure 1 The figure shows that the method includes steps S1, S2, S3, S4, S5 and S6.

[0032] Step S1: Obtain a close-up image of the hen's comb, a panoramic image of the hen's body, and a video of the hen moving.

[0033] Understandably, the close-up images of the comb in this step are taken using a fixed-focus lens under uniform lighting conditions to ensure that the distribution of microvessels and color gradations on the surface of the comb can be clearly captured; the panoramic images of the body are taken from a top-down angle to cover the complete feather distribution area of ​​the hen's back, wings and tail, providing spatial context information for subsequent feather damage analysis; the moving video is taken from a fixed position to continuously record the hen's natural activities over a specific period of time, focusing on capturing its walking gait, pecking frequency and other behavioral patterns.

[0034] This step uses close-up images of the crown to obtain biological features reflecting blood circulation status; panoramic images of the body provide spatial information for assessing the integrity of feather coverage; and moving videos record temporal data reflecting health status. This multi-dimensional data acquisition scheme lays the foundation for subsequent cross-modal feature fusion. By mutually verifying biological features from different dimensions, it overcomes the risk of misjudgment that may exist with a single data source, providing complete data support for a comprehensive assessment of the health status of laying hens.

[0035] Step S2: Based on the close-up image of the hen's comb, perform feature extraction processing to obtain abnormal image features and texture abnormal image features of the hen's comb blood circulation status.

[0036] Understandably, this step transforms historical images into quantifiable feature extraction standards through comparative analysis, improving the accuracy and interpretability of feature extraction. Texture anomaly processing amplifies microscopic pathological changes through multi-feature fusion, enhancing the sensitivity for identifying early lesions. In this step, step S2 includes steps S21, S22, and S23.

[0037] Step S21: Perform feature extraction processing based on the close-up image of the coronary region, wherein the close-up image of the coronary region is compared and analyzed with multiple preset images of historical blood circulation states, and the image difference regions obtained from the comparison are used as image features of the coronary blood circulation state.

[0038] Understandably, this step first involves constructing a historical image database containing comb features under different health conditions. These images are labeled according to normal blood circulation and further numbered according to different lighting conditions, resulting in standardized images of rooster combs under normal blood circulation. During comparative analysis, pixel matching and image registration are performed to ensure that the image to be tested and the historical samples are in the same spatial coordinate system. This allows for the identification of distinguishing regions from the standardized images of rooster combs under normal blood circulation, which are then used as image features.

[0039] Step S22: Perform texture enhancement processing based on the image features of the coronary blood circulation status, wherein the standard deviation of the image features of the coronary blood circulation status and multiple preset historical blood circulation status images is determined by the local binary mode variance analysis method to obtain texture distinguishing image features;

[0040] It is understood that this step calculates the feature parameters of the gray-level co-occurrence matrix by performing image features of the coronal blood circulation state. The feature parameters of the gray-level co-occurrence matrix include contrast, correlation, and entropy. Then, the local binary mode variance analysis method is used to establish a 3×3 neighborhood with each pixel as the center, calculate the standard deviation of the binary mode in the neighborhood, and take the difference between the standard deviation and the standard deviation of multiple preset historical blood circulation state images. The pixels with the difference greater than a preset threshold are used as texture distinguishing image features.

[0041] Step S23: Based on the texture difference image features and the preset similarity calculation of historical coronal close-up images with texture direction disorder caused by local ischemia or tissue necrosis, images with similarity greater than the preset threshold are used as texture abnormality image features.

[0042] The texture differentiation image features and texture differentiation features in the historical case database are vectorized, and principal component analysis (PCA) is used to reduce the dimensionality of all texture features. Then, the cosine similarity between the feature vectors of the texture differentiation image features and the feature vectors of each historical coronal close-up image is calculated. Pixels with similarity greater than a preset threshold are considered as texture anomaly image features. This step achieves effective dimensionality reduction of high-dimensional texture features through PCA, preserving the most discriminative feature components. The similarity judgment mechanism significantly improves the diagnostic accuracy of pathological conditions such as local ischemia and tissue necrosis, providing a reliable texture feature basis for health status assessment.

[0043] Step S3: Based on the panoramic image of the body, locate the edge of the feather damage and the area where the parasites attach to the laying hen, and obtain a binary pathological feature map containing feather pecking marks and parasite distribution information.

[0044] Understandably, this step uses morphological gradients to locate the edges of feather damage and parasite attachment areas in laying hens, effectively identifying damage features at different scales, from macroscopic feather defects to microscopic feather breaks. Through an adversarial attention mechanism-based focused learning strategy, the detection sensitivity for tiny parasites against a complex feather background is significantly improved. Furthermore, a graph cut optimization algorithm, by minimizing the energy function, ensures the accuracy and integrity of the pathological region boundaries, providing a reliable binary pathological feature map for subsequent health status analysis. This hierarchical processing mechanism guarantees comprehensive and accurate assessment of feather pecking marks and parasite distribution. In this step, step S3 includes steps S31, S32, and S33.

[0045] Step S31: Perform morphological gradient edge detection processing based on the panoramic image of the body, wherein the physical contour features of the feather damage area are accurately extracted by the morphological gradient operator to obtain a binary image of the feather damage edge.

[0046] Understandably, this step first uses a circular structuring element with a radius of 3 pixels to perform dilation and erosion operations on the panoramic image of the body. The dilation operation uses the structuring element to perform local maximum filtering on the image, causing the edges of the feather damage area to expand outwards; the erosion operation performs local minimum filtering, causing the edges to shrink inwards. The difference between the dilation and erosion results is then calculated to obtain the initial edge intensity map. This gradient operation effectively highlights the boundary features of the feather breakage area, exposed skin area, and other normal areas. In actual processing, a grayscale threshold is set to 1.5 times the overall mean of the gradient image, converting the edge intensity map into a binary image. Specifically, each pixel is judged: when the gradient value is greater than the threshold, it is set to 1 (edge ​​point); otherwise, it is set to 0 (non-edge point). For example, when processing locally damaged feather areas, this method can accurately identify breakage edges with a minimum width of 2 pixels, while effectively suppressing interference from the internal details of the feather texture itself. In the final output binary image, white pixels represent the detected feather damage edges, and black pixels represent the background area. This step accurately captures the physical contour of the feather damage area through morphological gradient operations, achieving pixel-level edge localization accuracy.

[0047] Step S32: Perform adversarial attention segmentation processing based on the binary map of the feather damage edge. In this process, a generative adversarial network with attention gates is constructed to locate the parasites at the base of the feather and obtain a parasite distribution probability map.

[0048] Understandably, this step utilizes a generative adversarial network with attention gating to achieve parasite localization. The generator employs an encoder-decoder structure; the encoder extracts image features through successive convolutional layers, and an attention gating mechanism is introduced during the decoding stage. This mechanism calculates the association weights between each location in the feature map and the damaged feather area, causing the network to focus on key areas such as the feather root.

[0049] This step first uses the binary image of the feather damage edge as a spatial attention template, performing a dot product operation with the feature map to enhance the feature response of areas where parasites may attach. The last layer of the network uses the Sigmoid activation function, outputting a continuous value between 0 and 1 for each pixel, representing the probability of a parasite being present at that location. During training, the probability map output by the generator is compared with the ground truth labels to calculate a binary cross-entropy loss, while the discriminator distinguishes between true and false probabilities. Through alternating optimization, the generator can finally output the probability of parasite presence for each pixel. For example, the dark area at the base of the feather will generate a high probability value of 0.9 or higher, while the probability value for normal feather areas is lower than 0.1. This step outputs the results in the form of a probability map, preserving the uncertainty information of parasite detection and providing continuous confidence for subsequent decisions.

[0050] Step S33: Perform graph cut optimization and fusion processing based on the binary map of feather damage edge and the parasite distribution probability map. Specifically, by establishing an energy function minimization graph cut model, optimize the boundaries of feather damage and parasite attachment areas of laying hens and fuse their spatial topological relationships to obtain a binary pathological feature map.

[0051] Understandably, this step first constructs an energy function, which includes a data term and a smoothing term. The data term, based on the parasite distribution probability map, is the cost of labeling each pixel as either a pathological region (foreground) or a normal region (background). For a pixel, when the pixel label is 1 (pathological region), the data term cost is the probability value in the parasite probability map; when the pixel label is 0 (normal region), the cost is a logarithmic function of the probability value in the parasite probability map. The smoothing term, based on a binary map of feather damage edges, encourages spatially adjacent pixels to have the same label, but reduces the smoothing cost at the edges. Specifically, for adjacent pixel pairs, the smoothing cost is set as follows:

[0052] ;

[0053] Among them, V ij (l i ,l j ) represents the smoothing cost, l i The label representing pixel i, l j Let λ represent the label of pixel j, and λ represent the weight coefficient of the smoothing term. Let represent the gradient difference between pixel i and pixel j, and Q represent the parameter controlling the exponential decay rate. It represents the reciprocal of the spatial distance between pixel i and pixel j.

[0054] This step minimizes the total energy function using the maximum flow algorithm to obtain the optimized binary label. Finally, the edge of the feather damage is merged with the parasite region to generate a unified ternary pathological feature map (0: normal, 1: feather damage, 2: parasite region).

[0055] This step uses an energy function model combined with probability information and edge constraints to achieve accurate segmentation of feather damage and parasite areas. Graph cut optimization ensures the continuity and smoothness of the pathological area boundary and reduces isolated noise points.

[0056] Step S4: Analyze abnormal behavior features based on the video footage of the laying hens moving. Specifically, analyze the abnormal behavior features of the laying hens by reconstructing the error to obtain the spatiotemporal feature sequence of the abnormal behavior of the laying hens.

[0057] Understandably, this step first constructs a convolutional autoencoder network. The encoder uses 3D convolutional layers to extract spatiotemporal features, and the decoder reconstructs video frames through deconvolution. During the training phase, normal behavior videos are used to learn reconstruction capabilities, and during the testing phase, the reconstruction error for each frame is calculated. Regions with errors exceeding a threshold are marked as abnormal. This step effectively captures abnormal behavior patterns through reconstruction error analysis, improving the sensitivity of detecting abnormal behavior in laying hens. Step S4 in this step includes steps S41, S42, and S43.

[0058] Step S41: Perform convolutional autoencoder reconstruction processing on the video of the hen's movement. The network is trained to learn the spatiotemporal feature representation in the normal behavior mode, and the difference pixel blocks between the original frame and the reconstructed frame are determined to obtain the behavior reconstruction error map.

[0059] Understandably, this step first constructs a deep network with an encoder-decoder structure. The encoder uses three consecutive 3D convolutional layers with 3×3×3 kernels and a stride of 2 for downsampling, progressively extracting the spatiotemporal features of the video sequence. The decoder uses a symmetrical structure, progressively recovering video frames through 3D deconvolution operations. During network training, preset normal behavior video clips are used, and network parameters are optimized by minimizing the reconstruction loss function, which uses the mean squared error calculation formula. After training, the network has the ability to reconstruct normal behavior patterns. When processing new videos, consecutive frames (e.g., 16 frames) are used as input, and the network outputs the reconstructed data. By comparing the differences between the original frames and the reconstructed frames pixel by pixel, the error value at each pixel position is calculated, forming an error map. The specific calculation method is shown below:

[0060] ;

[0061] Where E(i,j) represents the reconstruction error at pixel position (i,j). This represents the pixel value at position (i,j) in the original image. This represents the pixel value at position (i,j) in the reconstructed image. This indicates absolute value calculation.

[0062] In this step, a 3D convolutional autoencoder is used to learn normal behavior patterns, the principle of reconstruction error is used to identify abnormal behavior regions, and a block detection strategy is adopted to improve the sensitivity to local abnormal motion, so as to provide reliable abnormal region localization information for subsequent time series analysis.

[0063] Step S42: Perform time-series anomaly analysis based on the behavior reconstruction error map, wherein anomaly points in the error map are identified by applying a peak detection algorithm, and the anomaly persistence characteristics are analyzed by combining a time sliding window to obtain preliminary abnormal behavior segments.

[0064] Understandably, this step first employs an adaptive peak detection algorithm to perform temporal statistics on the number of abnormal pixel blocks in each frame's error map, forming an error value sequence. Next, threshold analysis is performed by setting a dynamic threshold; when the error value exceeds the threshold for three consecutive frames, it is marked as an anomaly. Then, a continuous analysis is conducted using a sliding window of 2 seconds (with a 50% overlap), calculating the proportion of abnormal frames, the integral value of the anomaly intensity, and the rate of change of the anomaly pattern within each window.

[0065] The formula for setting the dynamic threshold is as follows:

[0066] ;

[0067] Where θ represents the dynamic threshold, μ is the sequence mean, and P is the standard deviation.

[0068] Finally, when the proportion of abnormal frames within a window exceeds 60% and the abnormality intensity continues to increase, a significant anomaly is determined to exist in that time period. Finally, a time-series clustering algorithm is used to merge adjacent abnormal windows into continuous segments, obtaining preliminary abnormal behavior segments. This step achieves precise anomaly point localization through peak detection, utilizes sliding window analysis to ensure the persistence of abnormal behavior, and uses time-series clustering to generate complete abnormal behavior segments, providing reliable temporal localization information for subsequent feature extraction.

[0069] Step S43: Perform attention LSTM encoding processing based on the preliminary abnormal behavior fragments. The abnormal behavior features within the fragments are extracted through a long short-term memory network, and the feature representations of important time steps are weighted using an attention mechanism to obtain the spatiotemporal feature sequence of abnormal behavior of laying hens.

[0070] Understandably, this step first inputs the video frame sequence of each aberration segment into the LSTM network. The LSTM's input gate, forget gate, and output gate control the flow of information, with the input gate determining the acceptance level of the current input, as shown below:

[0071] ;

[0072] Among them, i t Here, σ represents the input gate state of the LSTM, and W represents the sigmoid activation function. i This represents the weight matrix of the input gate. This means combining the hidden state h(t-1) from the previous time step with the current input x. t To splice, b i This represents the bias vector of the input gate.

[0073] The forget gate controls the percentage of historical information retained, as shown below:

[0074] ;

[0075] Among them, f t Let W be the forget gate state of the LSTM, σ represent the sigmoid activation function, and W be the value of the sigmoid activation function. f The weight matrix represents the forget gate. This means combining the hidden state h(t-1) from the previous time step with the current input x. t To splice, b f This represents the bias vector of the forget gate.

[0076] The output gate adjusts the output intensity of the current state, as shown below:

[0077] ;

[0078] Among them, f o Let W be the forget gate state of the LSTM, σ represent the sigmoid activation function, and W be the value of the sigmoid activation function. o This represents the weight matrix of the output gate. This means combining the hidden state h(t-1) from the previous time step with the current input x. t To splice, b o This represents the bias vector of the output gate.

[0079] At each time step, the LSTM unit outputs the hidden state, containing behavioral feature information from the current moment and past times. An attention mechanism is then introduced to calculate the attention weights for each time step, as shown in the following formula:

[0080] ;

[0081] Where, α t The attention weights are for time step t. Represents the normalized exponential function, The transpose of a learnable parameter vector. Let H represent the hyperbolic tangent activation function, W represent the learnable weight matrix, and h represent the hyperbolic tangent activation function. t Let represent the hidden state vector at time step t, and b represent the learnable bias vector.

[0082] The final feature representation is a weighted sum of the hidden states at each time step, forming a fixed-dimensional spatiotemporal feature vector. This step uses an LSTM network to capture the temporal dependencies of abnormal behaviors and leverages an attention mechanism to highlight the feature contributions of key time steps, generating a discriminative spatiotemporal feature representation that provides effective behavioral feature input for subsequent health status assessment.

[0083] Step S5: Based on the abnormal image features, texture abnormal image features, binarized pathological feature map and abnormal behavior spatiotemporal feature sequence of the blood circulation status of the coronary of the laying hen, feature interaction processing is performed to construct a health status map with pathological regions as nodes and spatiotemporal correlations as edges.

[0084] It is understandable that this step achieves the organic integration of multimodal features through graph structure, establishes the intrinsic relationship between features using spatiotemporal association rules, and enhances the feature representation capability through graph convolution, providing a structured and interpretable feature representation for health status assessment. In this step, step S5 includes steps S51, S52, and S53.

[0085] Step S51: Based on the hypergraph modeling method, the abnormal image features, texture abnormal image features, binarized pathological feature map and abnormal behavior spatiotemporal feature sequence of the coronary blood circulation status of the laying hen are abstracted into hypernodes, and hyperedge connections are established based on spatial co-occurrence relationship to obtain a preliminary multimodal feature hypergraph.

[0086] This step, as is understandable, first involves normalizing the four feature vectors. For the abnormal blood circulation feature of the crown, the hue component statistical histogram (16-dimensional) of the HSV color space is extracted. For the abnormal texture feature, the binarized texture feature vector (256-dimensional) is calculated. For the binarized pathological feature map, the Zernike moment feature (12-dimensional) is extracted. The spatiotemporal feature of abnormal behavior uses a 128-dimensional vector output from an LSTM. These feature vectors are then combined to form a supernode feature representation. Each supernode contains a concatenated vector of the above four types of features. The construction of hyperedges is based on the spatial co-occurrence relationship between features. Specifically, if the abnormal crown region and the feather damage region overlap spatially (overlap area ratio > 30%), then the supernodes corresponding to these regions are connected by a hyperedge. If an abnormal behavior and a specific pathological feature occur synchronously in time (time synchronization window ± 5 seconds), then a spatiotemporal hyperedge is established. The weight of the hyperedge is determined by the feature similarity and spatial distance between nodes, calculated as follows:

[0087] ;

[0088] in, Let be the weight of the hyperedge e, exp represent the exponential function, d represent the spatial distance, and S represent the distance scaling parameter (hyperparameter). Represents the eigenvector f i and eigenvector f j The similarity function.

[0089] This step achieves deep fusion of multimodal features through a hypergraph structure, establishes semantic associations between features using spatial co-occurrence relationships, and expresses the intensity differences of feature interactions through weighted hyperedges, providing rich structured feature representations for subsequent health status analysis. The hypergraph model can effectively capture complex many-to-many feature relationships and better reflects the multi-factor correlation characteristics of health status than traditional graph structures.

[0090] Step S52: Calculate the feature association weights between pairs of supernodes in the preliminary multimodal feature hypergraph based on the graph attention network mechanism to obtain the weighted health status map;

[0091] Understandably, this step uses a graph attention network to adaptively calculate the association weights between supernodes. First, a linear transformation is performed on the features of each supernode, projecting them into a new feature space. Then, a self-attention mechanism is used to calculate the association strength between nodes: for each central node, its attention coefficient with all neighboring nodes is calculated. This coefficient is obtained by connecting the features of two nodes and passing them through a single-layer feedforward network. Specifically, the central node features are concatenated with the neighboring node features, transformed by a weight matrix, and then activated using the sigmoid activation function to obtain an unnormalized attention score. Next, a normalized exponential function is used to standardize the attention score, ensuring that the sum of all weights is 1. A multi-head attention mechanism is employed, running multiple independent attention calculation processes in parallel. The feature representations obtained from different attention heads are concatenated or averaged, thereby stabilizing the learning process and enhancing the model's expressive power. Finally, the calculated attention weights are weighted and summed with the feature vectors of the corresponding nodes, and the feature representation of each node is updated to obtain a weighted health status map.

[0092] This step highlights important feature interactions through attention mechanisms, enhances feature representation capabilities by utilizing multi-head attention, and ultimately generates a weighted health status map that accurately reflects the intrinsic relationships between features.

[0093] Step S53: Perform dynamic temporal graph convolution processing based on the weighted health status graph, wherein the graph features of adjacent time frames are aggregated by a gated graph neural network to capture the temporal change features of the health status and obtain a dynamic health status graph.

[0094] Understandably, this step first organizes the weighted health status graphs of consecutive time periods into a graph sequence in chronological order. Each time step's graph contains node features and attention weights. A gated graph neural network unit is used to process this sequence. This unit includes update and reset gates, which control the retention rate of historical information and the integration of new information, respectively. Specifically, for each time step, a graph convolution operation is first performed to aggregate neighbor node information. The specific operation formula is as follows:

[0095] ;

[0096] in, Show the updated feature representation of node I. h represents the attention weight of node I's neighbor node J. J This represents the feature vector of neighbor node J. This represents the set of neighboring nodes of node I.

[0097] Then, the graph convolution result of the current time step and the hidden state of the previous time step are input into the gating unit to update the gating unit, calculate the adoption ratio of new information, and reset the gating unit to control the forgetting degree of historical information. Finally, the output features of multiple time steps are passed through a one-dimensional convolutional layer in the time dimension, and a temporal convolutional kernel of length 3 is used to capture short-term temporal patterns.

[0098] This step achieves adaptive fusion of time-series information through a gating mechanism, effectively capturing the dynamic changes in health status. The generated dynamic health status map contains time-series dimension information, providing a data foundation for health trend prediction.

[0099] Step S6: Based on the health status map, perform time series analysis and health score processing, wherein the contribution of all features is quantified through graph attention mechanism to obtain a diagnostic report containing specific disease risk labels and health scores.

[0100] It is understandable that this step achieves a quantitative assessment of health status through a time-series graph attention mechanism. This mechanism dynamically quantifies the importance of features, accurately reflecting the changes in the contribution of each indicator over time. Step S6 in this step includes steps S61, S62, and S63.

[0101] Step S61: Perform dynamic feature weight calculation processing based on the health status map, wherein the temporal change pattern of node features is analyzed through the time sequence graph attention mechanism, the importance weight of all features at each time point is quantified, and a dynamic feature weight matrix is ​​obtained.

[0102] Understandably, this step achieves dynamic quantization of feature weights through a temporal graph attention network. First, a time-sliding window is constructed to organize the health status graphs of consecutive time slices into a temporal graph sequence. For each time step, a multi-head graph attention mechanism is used to calculate the importance weights of node features. The specific process includes: performing a linear transformation on each node feature to generate a query vector, key vector, and value vector; calculating the dot product similarity between the query vector and the key vectors of all adjacent nodes; applying a time decay factor to adjust the attention weights of historical time steps, assigning higher weights to recent time steps; and finally, normalization using a softmax function. A time encoding mechanism is introduced during the weight calculation process to embed time interval information into the attention calculation.

[0103] The attention score is calculated as follows:

[0104] ;

[0105] Where A represents the attention score. This represents vector concatenation, where σ is the sigmoid function, T represents the time-encoded vector, q represents the query vector, k represents the key vector, and ● represents the vector dot product operation. Multiple sets of attention weights are computed in parallel using a multi-head attention mechanism to capture feature dependencies at different time scales.

[0106] This step dynamically captures the temporal variation patterns of feature importance through a time-series graph attention mechanism, adapting to the dynamic evolution of health status; it utilizes a multi-head attention mechanism to extract diverse temporal dependencies from different feature subspaces, enhancing the model's expressive power; and it combines a time decay factor to ensure that recent features receive higher attention, improving sensitivity to changes in health status; the resulting dynamic feature weight matrix provides interpretable quantitative evidence for health scores, achieving accurate quantification of feature contribution.

[0107] Step S62: Perform health index calculation based on the dynamic feature weight matrix, wherein multimodal feature weights are integrated through a weighted fusion algorithm and combined with a time-series aggregation function to generate a health index curve for a continuous time period.

[0108] Understandably, this step generates a health index curve through multimodal feature fusion and temporal analysis. First, the dynamic feature weight matrix is ​​normalized to ensure the sum of all feature weights is 1. Then, a weighted fusion algorithm is used to multiply the coronary blood circulation features, texture features, pathological features, and behavioral features by their corresponding temporal weights to calculate the initial health index value at each time point. The specific calculation formula is shown below:

[0109] ;

[0110] in, This represents the weight of the i-th feature at time t. This represents the value of the i-th feature at time t, where t represents time, i represents the i-th feature, and n represents the total number of features.

[0111] Then, a time-series aggregation function was used to smooth the initial values ​​of the health index, and a moving average algorithm was employed to eliminate random fluctuations. The window size was set to 5 time units. A time decay factor was also introduced, assigning higher weights to recent data and decreasing weights to historical data over time. Finally, a continuous health index curve was generated using cubic spline interpolation to ensure the curve was smooth and reflected the continuous trend of health status changes.

[0112] Step S63: Perform disease risk classification processing based on the health index curve. This involves performing correlation analysis between the health index curve and preset historical disease curve data of laying hens, and using the correlation degree as the risk value of the laying hen for each disease, thereby obtaining a health diagnosis report containing disease risk labels.

[0113] Understandably, this step achieves quantitative assessment of disease risk through historical curve matching. First, a database of historical health index curves containing common diseases such as avian influenza is established, with each curve labeled with the disease type and severity. A dynamic time warping algorithm is used to calculate the similarity between the current health index curve and the template curves for each disease. By finding the optimal time alignment path, the influence of individual differences and monitoring time offsets is eliminated. Based on the similarity results, risk probability estimation is performed, using a Bayesian classifier to calculate the posterior probability that the current curve belongs to each disease category. Risk level thresholds are set: similarity greater than 0.9 indicates high risk, 0.7-0.9 indicates medium risk, and 0.5-0.7 indicates low risk. Finally, a structured diagnostic report is generated, including risk values ​​for each disease, descriptions of key abnormal features, and prevention recommendations.

[0114] Example 2:

[0115] like Figure 2 As shown, this embodiment provides a multimodal image recognition-based system for monitoring the health status of laying hens. (See also...) Figure 2 The system includes an acquisition unit 701, an extraction unit 702, a positioning unit 703, an analysis unit 704, a construction unit 705, and a diagnosis unit 706.

[0116] The acquisition unit 701 is used to acquire close-up images of the hen's comb, panoramic images of the hen's body, and video footage of the hen's movement.

[0117] Extraction unit 702 is used to perform feature extraction processing based on the close-up image of the hen's crown to obtain abnormal image features and texture abnormal image features of the hen's crown blood circulation status.

[0118] The positioning unit 703 is used to locate the edge of feather damage and the area where parasites attach to the laying hen based on the panoramic image of the body, and to obtain a binary pathological feature map containing feather pecking marks and parasite distribution information.

[0119] Analysis unit 704 is used to perform abnormal behavior feature analysis based on the video of the movement of laying hens. The abnormal behavior features of laying hens are analyzed by reconstruction error analysis to obtain the spatiotemporal feature sequence of abnormal behavior of laying hens.

[0120] The construction unit 705 is used to perform feature interaction processing based on the abnormal image features, texture abnormal image features, binarized pathological feature map and abnormal behavior spatiotemporal feature sequence of the blood circulation status of the coronary of the laying hen, and to construct a health status map with pathological regions as nodes and spatiotemporal associations as edges.

[0121] The diagnostic unit 706 is used to perform time series analysis and health score processing based on the health status map, wherein the contribution of all features is quantified through a graph attention mechanism to obtain a diagnostic report containing specific disease risk labels and health scores.

[0122] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0124] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for monitoring the health status of laying hens based on multimodal image recognition, characterized in that, include: Acquire close-up images of the hens' combs, panoramic images of the hens' bodies, and videos of the hens moving. Based on the close-up image of the hen's comb, feature extraction processing was performed to obtain abnormal image features and texture abnormal image features of the hen's comb blood circulation status. Based on the panoramic image of the body, the edges of feather damage and the areas where parasites attach to the laying hen are located, resulting in a binary pathological feature map containing feather pecking marks and parasite distribution information. Abnormal behavior feature analysis is performed based on video footage of laying hens moving around. Specifically, abnormal behavior features of laying hens are obtained by analyzing reconstruction error, resulting in a spatiotemporal feature sequence of abnormal behavior of laying hens. Based on the abnormal image features, texture abnormal image features, binarized pathological feature map and abnormal behavior spatiotemporal feature sequence of the blood circulation status of the coronary of the laying hen, feature interaction processing is performed to construct a health status map with pathological regions as nodes and spatiotemporal correlations as edges. Based on the health status map, time series analysis and health score processing are performed. The contribution of all features is quantified through graph attention mechanism to obtain a diagnostic report that includes specific disease risk labels and health scores. The feature interaction processing, based on abnormal image features of the blood circulation status of the hen's crown, abnormal texture image features, binarized pathological feature maps, and spatiotemporal feature sequences of abnormal behavior, includes: Based on the hypergraph modeling method, the abnormal image features, texture abnormal image features, binarized pathological feature maps and spatiotemporal feature sequences of abnormal behavior of the coronary blood circulation status of the laying hen are abstracted into hypernodes, and hyperedge connections are established based on spatial co-occurrence relationships to obtain a preliminary multimodal feature hypergraph. The feature association weights between pairs of supernodes in the preliminary multimodal feature hypergraph are calculated based on the graph attention network mechanism to obtain the weighted health status map. Dynamic temporal graph convolution processing is performed based on the weighted health status graph, wherein the graph features of adjacent time frames are aggregated by a gated graph neural network to capture the temporal change features of health status and obtain a dynamic health status graph. Based on the health status map, time series analysis and health scoring are performed. The contribution of all features is quantified using a graph attention mechanism, including: Based on the health status map, dynamic feature weight calculation is performed. The temporal change pattern of node features is analyzed through the time sequence graph attention mechanism, and the importance weight of all features at each time point is quantified to obtain the dynamic feature weight matrix. The health index is calculated based on the dynamic feature weight matrix, wherein a weighted fusion algorithm is used to integrate multimodal feature weights and a time-series aggregation function is combined to generate a health index curve for a continuous time period. Disease risk classification is performed based on the health index curve. This involves correlation analysis between the health index curve and preset historical disease curve data of laying hens, and using the correlation degree as the risk value of the laying hen for each disease, thereby obtaining a health diagnosis report containing disease risk labels.

2. The method for monitoring the health status of laying hens based on multimodal image recognition according to claim 1, characterized in that... Feature extraction processing is performed based on the close-up image of the hen's comb, including: Feature extraction processing is performed based on the close-up image of the coronary region. Specifically, the close-up image of the coronary region is compared and analyzed with multiple preset images of historical blood circulation states, and the image difference regions obtained from the comparison are used as image features of the coronary blood circulation state. Texture enhancement processing is performed based on the image features of the coronary blood circulation status. Specifically, the standard deviation of the image features of the coronary blood circulation status and multiple preset historical blood circulation status images is determined by the local binary mode variance analysis method to obtain texture distinguishing image features. Based on texture-distinguishing image features and a preset similarity calculation of historical coronal close-up images with texture direction disorder caused by local ischemia or tissue necrosis, images with similarity greater than a preset threshold are used as texture abnormality image features.

3. The method for monitoring the health status of laying hens based on multimodal image recognition according to claim 1, characterized in that... Based on the panoramic image of the body, the edges of feather damage and parasite attachment areas of the laying hen are located, resulting in a binary pathological feature map containing feather pecking marks and parasite distribution information; including: Based on the panoramic image of the body, morphological gradient edge detection processing is performed, wherein the physical contour features of the feather damage area are accurately extracted by the morphological gradient operator to obtain a binary image of the feather damage edge. Adversarial attention segmentation is performed based on the binary map of the feather damage edge. Specifically, a generative adversarial network with attention gates is constructed to locate parasites at the base of the feather, thereby obtaining a parasite distribution probability map. Based on the binary map of feather damage edges and the probability map of parasite distribution, a graph cut optimization fusion process is performed. Specifically, by establishing a graph cut model that minimizes the energy function, the boundaries of feather damage and parasite attachment areas in laying hens are optimized and their spatial topological relationships are fused to obtain a binary pathological feature map.

4. The method for monitoring the health status of laying hens based on multimodal image recognition according to claim 1, characterized in that... This study analyzes abnormal behavior features of laying hens based on video footage of their movement. Specifically, it uses reconstruction error analysis to identify the abnormal behavior characteristics of the hens, resulting in a spatiotemporal feature sequence of their abnormal behavior, including: Based on the video footage of the hen's movement, a convolutional autoencoder is used for reconstruction. The network is trained to learn the spatiotemporal feature representation of the normal behavior pattern, and the difference pixel blocks between the original frame and the reconstructed frame are determined to obtain the behavior reconstruction error map. Based on the behavior reconstruction error map, a time-series anomaly analysis is performed. Specifically, anomaly points in the error map are identified by applying a peak detection algorithm, and the anomaly persistence characteristics are analyzed by combining a time sliding window to obtain preliminary abnormal behavior segments. Based on the preliminary abnormal behavior fragments, attention LSTM encoding is performed, wherein the abnormal behavior features within the fragments are extracted through a long short-term memory network, and the feature representations of important time steps are weighted using an attention mechanism to obtain the spatiotemporal feature sequence of abnormal behavior of laying hens.

5. A system for monitoring the health status of laying hens based on multimodal image recognition, characterized in that, include: The acquisition unit is used to acquire close-up images of the hen's comb, panoramic images of the hen's body, and video footage of the hen's movement. The extraction unit is used to perform feature extraction processing based on the close-up image of the hen's crown to obtain abnormal image features and texture abnormal image features of the hen's crown blood circulation status. The positioning unit is used to locate the edge of feather damage and the area where parasites attach to the laying hen based on the panoramic image of the body, and to obtain a binary pathological feature map containing feather pecking marks and parasite distribution information. The analysis unit is used to perform abnormal behavior feature analysis based on the video of the movement of laying hens. Specifically, the abnormal behavior features of laying hens are analyzed by reconstruction error analysis to obtain the spatiotemporal feature sequence of abnormal behavior of laying hens. The construction unit is used to perform feature interaction processing based on the abnormal image features, texture abnormal image features, binarized pathological feature map and abnormal behavior spatiotemporal feature sequence of the blood circulation status of the coronary of the laying hen, and to construct a health status map with pathological regions as nodes and spatiotemporal associations as edges. The diagnostic unit is used to perform time series analysis and health score processing based on the health status map, wherein the contribution of all features is quantified through a graph attention mechanism to obtain a diagnostic report containing specific disease risk labels and health scores. The feature interaction processing, based on abnormal image features of the blood circulation status of the hen's crown, abnormal texture image features, binarized pathological feature maps, and spatiotemporal feature sequences of abnormal behavior, includes: Based on the hypergraph modeling method, the abnormal image features, texture abnormal image features, binarized pathological feature maps and spatiotemporal feature sequences of abnormal behavior of the coronary blood circulation status of the laying hen are abstracted into hypernodes, and hyperedge connections are established based on spatial co-occurrence relationships to obtain a preliminary multimodal feature hypergraph. The feature association weights between pairs of supernodes in the preliminary multimodal feature hypergraph are calculated based on the graph attention network mechanism to obtain the weighted health status map. Dynamic temporal graph convolution processing is performed based on the weighted health status graph, wherein the graph features of adjacent time frames are aggregated by a gated graph neural network to capture the temporal change features of health status and obtain a dynamic health status graph. Based on the health status map, time series analysis and health scoring are performed. The contribution of all features is quantified using a graph attention mechanism, including: Based on the health status map, dynamic feature weight calculation is performed. The temporal change pattern of node features is analyzed through the time sequence graph attention mechanism, and the importance weight of all features at each time point is quantified to obtain the dynamic feature weight matrix. The health index is calculated based on the dynamic feature weight matrix, wherein a weighted fusion algorithm is used to integrate multimodal feature weights and a time-series aggregation function is combined to generate a health index curve for a continuous time period. Disease risk classification is performed based on the health index curve. This involves correlation analysis between the health index curve and preset historical disease curve data of laying hens, and using the correlation degree as the risk value of the laying hen for each disease, thereby obtaining a health diagnosis report containing disease risk labels.

6. The laying hen health status monitoring system based on multimodal image recognition according to claim 5, characterized in that, The extraction unit includes: The first extraction subunit is used to perform feature extraction processing based on the close-up image of the coronary region. The close-up image of the coronary region is compared and analyzed with multiple preset images of historical blood circulation states, and the image difference regions obtained by comparison are used as image features of the coronary blood circulation state. The second extraction subunit is used to perform texture enhancement processing based on the image features of the coronary blood circulation status. The standard deviation of the image features of the coronary blood circulation status and multiple preset historical blood circulation status images is determined by the local binary mode variance analysis method to obtain texture distinguishing image features. The third extraction subunit is used to calculate the similarity between the texture-distinguishing image features and the preset historical close-up images of the coronal region with texture direction disorder caused by local ischemia or tissue necrosis. Images with similarity greater than a preset threshold are used as texture abnormality image features.

7. The laying hen health status monitoring system based on multimodal image recognition according to claim 5, characterized in that, The positioning unit includes: The first positioning subunit is used to perform morphological gradient edge detection processing based on the panoramic image of the body, wherein the physical contour features of the feather damage area are accurately extracted by the morphological gradient operator to obtain a binary image of the feather damage edge. The second localization subunit is used to perform adversarial attention segmentation processing based on the binary map of the feather damage edge. Specifically, by constructing a generative adversarial network with attention gates, the parasites at the base of the feather are located to obtain a parasite distribution probability map. The third positioning subunit is used to perform graph cut optimization fusion processing based on the binary map of feather damage edge and the parasite distribution probability map. Specifically, by establishing an energy function minimization graph cut model, the boundaries of feather damage and parasite attachment areas of laying hens are optimized and their spatial topological relationships are fused to obtain a binary pathological feature map.

8. The laying hen health status monitoring system based on multimodal image recognition according to claim 5, characterized in that, The analysis unit includes: The first analysis subunit is used to perform convolutional autoencoder reconstruction processing based on the video of the hen's movement. The network is trained to learn the spatiotemporal feature representation of the normal behavior pattern, and the difference pixel blocks between the original frame and the reconstructed frame are determined to obtain the behavior reconstruction error map. The second analysis subunit is used to perform time-series anomaly analysis based on the behavior reconstruction error map. The anomaly points in the error map are identified by applying a peak detection algorithm, and the anomaly persistence characteristics are analyzed by combining a time sliding window to obtain preliminary abnormal behavior segments. The third analysis subunit is used to perform attention LSTM encoding processing based on the preliminary abnormal behavior fragments. The abnormal behavior features within the fragments are extracted through a long short-term memory network, and the feature representations of important time steps are weighted using an attention mechanism to obtain the spatiotemporal feature sequence of abnormal behavior of laying hens.

Images