A method and system for health assessment of squab based on multi-modal data fusion

Abstract

This invention discloses a method and system for assessing pigeon health based on multimodal data fusion, relating to the field of livestock and poultry breeding monitoring technology. The method includes: acquiring and preprocessing multimodal video sequences of a target pigeon cage to obtain multimodal data frames; acquiring multidimensional spatial features of the pigeon's head based on the multimodal data frames; performing identity association matching on individual pigeons in consecutive frames based on the multimodal spatial features to generate temporal trajectories; constructing a temporal observation feature sequence based on the temporal trajectories and extracting the behavioral classification results and behavioral duration features of the corresponding pigeon individuals as behavioral data; acquiring the infrared observation temperature of the pigeon's head based on the multimodal data frames and performing temperature compensation to obtain the pigeon's core body temperature; and fusing behavioral data, pigeon core body temperature, and environmental parameters to obtain health status features and acquire pigeon health assessment grading results. This achieves accurate measurement of the pigeon's core body temperature, reliable behavioral identification, and comprehensive health assessment of multidimensional information.

Description

Technical Field

[0001] This invention relates to the field of livestock and poultry breeding monitoring technology, and more specifically to a method and system for assessing the health of squabs based on multimodal data fusion. Background Technology

[0002] In group rearing, high stocking densities of squabs lead to rapid disease transmission. Currently, most pigeon farms still rely primarily on manual inspections for health monitoring. This method is labor-intensive, inefficient, and the frequent entry of inspectors into the pigeon lofts can easily trigger stress in the flock. Furthermore, manual observation has a certain time lag, making it difficult to detect individual abnormalities promptly. With the development of non-contact monitoring technology, automated monitoring methods based on vision and sensors are gradually being applied in poultry farming.

[0003] However, in high-density cage rearing environments, the relevant technologies still have the following shortcomings: First, due to the dense distribution of squabs and frequent occlusion, existing target tracking methods based on two-dimensional images mainly rely on pixel spatial location or appearance features for matching, which is prone to target identity switching or tracking interruption, making it difficult to obtain continuous and stable individual temporal trajectory data. Second, in terms of body temperature measurement, existing infrared thermometry methods mostly estimate the surface temperature observed in images, without fully considering the impact of changes in spatial distance between the target and the sensor and posture differences on heat radiation transmission, resulting in deviations between the measurement results and the actual core body temperature. Third, in terms of behavior recognition, existing methods mostly analyze two-dimensional image features, which are insufficient for distinguishing subtle behavioral patterns and making it difficult to accurately identify subtle changes in the squab's head during similar behaviors such as feeding and resting, thus affecting the accuracy of behavior recognition results. Fourth, in terms of health assessment, existing systems usually make judgments based on a single or few indicators, lacking the ability to comprehensively analyze behavioral characteristics, physiological parameters, and environmental factors, which can easily lead to misjudgments in complex breeding environments and make it difficult to achieve graded assessment of individual health status.

[0004] Therefore, how to accurately measure the core body temperature of squabs in high-density cage rearing environments, reliably identify fine-grained behaviors, and comprehensively assess the health of squabs using multi-dimensional information are problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a method and system for assessing the health of squabs based on multimodal data fusion, which overcomes or at least partially solves the above problems, and realizes accurate measurement of squab core body temperature, reliable identification of fine-grained behavior, and comprehensive health assessment of multidimensional information in high-density cage-rearing environment.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a method for assessing the health of squabs based on multimodal data fusion, comprising: The multimodal video sequence of the target pigeon cage is acquired and preprocessed to obtain multimodal data frames; The multidimensional spatial features of the pigeon's head are obtained based on the multimodal data frames. Based on the multidimensional spatial features, identity association matching is performed on individual pigeons in consecutive frames to generate time-series trajectories; Based on the time-series trajectory and the corresponding multidimensional spatial features, a time-series observation feature sequence is constructed, and the behavioral classification results and behavioral duration features of the corresponding pigeon individuals are extracted as behavioral data. The infrared observation temperature of the pigeon's head is obtained based on the multimodal data frame, and temperature compensation is performed to obtain the pigeon's core body temperature. Based on the behavioral data, the squab's core body temperature, and environmental parameters, health status characteristics are obtained, and the squab health assessment and grading results are acquired.

[0008] The preferred method for acquiring multimodal data frames is as follows: The multimodal video sequence includes: visible light images, depth images, and infrared thermal images; Obtain the camera calibration parameters of the corresponding acquisition device for the multimodal video sequence; Based on the camera calibration parameters, the depth image is reprojected onto the coordinate system of the visible light image to obtain an aligned depth map; The homography matrix is ​​obtained based on the feature point matching relationship between the visible light image and the infrared thermal image; Geometric correction is performed on the infrared thermal image based on the homography matrix to obtain an aligned infrared thermal image; Based on the synchronization timestamps of the data collected by each acquisition device, the visible light image, the aligned depth map, and the aligned infrared thermal image are synchronized in time and matched at the frame level. The visible light image, the aligned depth map, and the aligned infrared thermal image, which are aligned in both time and space, are fused to construct the multimodal data frame in a unified coordinate system.

[0009] Preferably, the method for obtaining the multidimensional spatial features is as follows: Based on the visible light image input to the convolutional neural network, the bounding box parameters of each pigeon's head are extracted; The bounding box region of the corresponding pigeon's head is determined based on the bounding box parameters. Based on the bounding box region, extract the coordinates of key points on the pigeon's head; The bounding box region is mapped onto the alignment depth map, and the effective mean depth of the bounding box covered area is extracted as the physical distance of the pigeon's head relative to the acquisition device. Based on the camera calibration parameters, the alignment depth map, and the key point coordinates, the three-dimensional attitude deflection angle of the pigeon's head is obtained; The multidimensional spatial features are obtained by fusing the bounding box parameters, the key point coordinates, the physical distance, and the three-dimensional attitude deflection angle.

[0010] Preferably, the method for obtaining the three-dimensional attitude deflection angle is as follows: Based on the alignment depth map and the camera intrinsic parameter matrix in the camera calibration parameters, the coordinates of the key points are back-projected in three dimensions to obtain the three-dimensional spatial coordinates of the beak tip position, the three-dimensional spatial coordinates of the left eye position and the three-dimensional spatial coordinates of the right eye position. Construct a direction vector for the line connecting the two eyes based on the three-dimensional spatial coordinates of the left eye position and the three-dimensional spatial coordinates of the right eye position; Based on the three-dimensional spatial coordinates of the beak tip position, the three-dimensional spatial coordinates of the left eye position, and the three-dimensional spatial coordinates of the right eye position, a sagittal direction vector from the beak tip to the center of both eyes is constructed; The spatial normal vector of the pigeon's head is obtained based on the direction vector of the eye-to-eye line and the direction vector of the sagittal plane. The angle between the spatial normal vector and the optical axis direction vector of the acquisition device is calculated to obtain the three-dimensional attitude deflection angle of the pigeon's head.

[0011] Preferably, generating time-series trajectories specifically includes: The multidimensional spatial matching distance is based on the matching distance between the multidimensional spatial features corresponding to any two different pigeons in adjacent frames. Based on the multidimensional spatial matching distance as the association cost, construct the association cost matrix between all pigeon individuals in the current frame and the previous frame; Based on the aforementioned correlation cost matrix, the Hungarian algorithm is used for global optimal matching; Determine whether the association cost corresponding to the successfully matched target pair is less than a preset association threshold; If so, the corresponding target is determined to belong to the same individual pigeon, and its unique identity is inherited; Otherwise, it is identified as a new target and a new identifier is assigned; The multidimensional spatial features corresponding to individual pigeons with the same identity in consecutive video frames are concatenated in chronological order to generate the temporal trajectory of the corresponding individual pigeon.

[0012] The preferred method for constructing time-series observation feature sequences is as follows: Based on the time-series trajectory, the multidimensional spatial feature changes of the same squab individual between adjacent time frames are obtained as micro-motion residuals. Based on the fusion of the micro-motion residuals and the corresponding multi-dimensional spatial features, a time-series observation vector characterizing the dynamic behavior of squabs is constructed. The temporal observation feature sequence of an individual pigeon is constructed by arranging the temporal observation vectors in continuous time frames in chronological order.

[0013] Preferably, the method for acquiring behavioral data is as follows: Based on the input of the time-series observation feature sequence into the micro-motion enhanced state space network, the time-series behavior of individual pigeons is modeled to obtain a historical state information vector. When the target is occluded, the target position is predicted based on the historical state information vector to maintain the continuity of the target identity sequence. At the same time, the historical state information vector is fused with the temporal observation vector to obtain the behavioral features. Based on the behavioral characteristics, a probability distribution of individual pigeons in each behavioral category is generated, and the behavioral category with the highest probability is selected as the behavioral classification result according to the probability distribution. The number of consecutive frames of the behavior classification results within a continuous time window is counted, and the duration feature of the behavior is obtained by combining the sampling frame rate. The behavior classification results and the behavior duration features together constitute the behavior data.

[0014] The preferred method for obtaining the core body temperature of squabs is as follows: The infrared observation temperature of the pigeon's head is extracted based on the aligned infrared thermogram in the multimodal data frame; A segmented thermal radiation compensation model is constructed based on a preset ranging boundary threshold, and the infrared observation temperature is compensated for distance according to the physical distance to obtain the distance-compensated temperature. The angular radiation attenuation factor is obtained based on the preset infrared emissivity coefficient of pigeon feathers and the three-dimensional attitude deflection angle. The core body temperature of the pigeon is obtained based on the distance-compensated temperature and the angular radiation attenuation factor.

[0015] Preferably, the results of the health assessment and grading of squabs are obtained, specifically including: Based on the synchronization timestamp, the core body temperature of the squab, the behavior classification results, the duration of the behavior, and the environmental parameters are time-aligned and resampled, and the health status feature vector is obtained by splicing the features of various preprocessed multi-source data. The health status feature vector is input into the random forest health assessment model to classify and assess the health status of individual pigeons, and the corresponding health assessment grade result of the individual pigeon is obtained. The random forest health assessment model is trained based on historical physiological data, behavioral category data, and veterinary clinical diagnostic sample data. It establishes a mapping relationship between the physiological state, behavioral patterns, and health status of pigeons through an integrated structure of multiple decision trees.

[0016] In a second aspect, embodiments of the present invention provide a pigeon health assessment system based on multimodal data fusion, used to execute a pigeon health assessment method based on multimodal data fusion as described in any of the first aspects, comprising: a data acquisition and processing module, a head feature extraction module, a time-series trajectory generation module, a behavioral data acquisition module, a core body temperature acquisition module, and an assessment result output module; The data acquisition and processing module is used to acquire the multimodal video sequence of the target pigeon cage and preprocess it to obtain multimodal data frames. The head feature extraction module is used to obtain multi-dimensional spatial features of the pigeon's head based on the multimodal data frame. The time-series trajectory generation module is used to perform identity association matching on individual pigeons in continuous frames based on the multi-dimensional spatial features to generate time-series trajectories. The behavior data acquisition module is used to construct a time-series observation feature sequence based on the time-series trajectory and the corresponding multi-dimensional spatial features, and extract the behavior classification results and behavior duration features of the corresponding pigeon individuals as behavior data. The core body temperature acquisition module is used to acquire the infrared observation temperature of the pigeon's head based on the multimodal data frame, and perform temperature compensation to obtain the pigeon's core body temperature. The assessment result output module is used to obtain health status characteristics based on the fusion of the behavioral data, the squab's core body temperature, and environmental parameters, and to obtain the squab's health assessment grading results.

[0017] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method and system for assessing the health of squabs based on multimodal data fusion, which has the following beneficial effects: 1. Improved target tracking stability in high-density aquaculture environments: This invention introduces physical distance and three-dimensional attitude deflection angle in the target association process, constructs multi-dimensional matching features for identity association, and combines a temporal state update mechanism to continuously track the target, thereby maintaining the continuity of the target identity even when individuals are occluded or overlapped, reducing identity switching and trajectory interruption phenomena.

[0018] 2. Improved accuracy of fine-grained behavior recognition: This invention extracts the spatial position and posture change information of pigeon head in continuous time frames, constructs micro-motion features and performs temporal modeling, thereby enhancing the ability to depict subtle differences in movements and improving the ability to distinguish similar behavior patterns.

[0019] 3. Improved stability of long-term time series behavior analysis: This invention models continuous time series based on state recursive update and combines micro-motion features for feature enhancement, thereby maintaining the continuity of key behavioral features during long-term time series analysis and improving the stability of behavior recognition results.

[0020] 4. Improved accuracy of core body temperature estimation: This invention introduces the physical distance between the target and the sensor and the three-dimensional attitude deflection angle during the infrared temperature measurement process to compensate and correct the infrared observed temperature, thereby reducing the impact of spatial attenuation and attitude changes on the temperature measurement results, making the body temperature estimation results closer to the actual core body temperature.

[0021] 5. Improved accuracy and reliability of health assessment: This invention performs time alignment and fusion processing on behavioral characteristics, body temperature characteristics and environmental parameters to construct health status characteristics, and conducts a comprehensive assessment based on these characteristics, thereby achieving graded judgment of the health status of individual pigeons and improving the reliability of monitoring results. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 This is a flowchart of a pigeon health assessment method based on multimodal data fusion provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of multimodal data acquisition and device mounting provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of a pigeon health assessment system based on multimodal data fusion provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1 like Figure 1 As shown, this invention discloses a method for assessing the health of squabs based on multimodal data fusion, comprising the following steps, which are numbered S1 to S6 for ease of description. These numbers are not intended to limit the sequential relationship between the various steps of this invention: S1 acquires the multimodal video sequence of the target pigeon cage and preprocesses it to obtain multimodal data frames.

[0028] Furthermore, before performing step S1 above, it is necessary to pre-construct the datasets required by this method, including the object detection dataset D1, the micro-movement behavior recognition dataset D2, and the health decision dataset D3. The specific construction process includes the following steps: like Figure 2 As shown, the automated inspection robot is controlled to perform automated fixed-distance navigation and acquire multimodal video sequences. Figure 2 As shown in section (b), the inspection robot is equipped with an integrated multimodal sensor gimbal, including: an AstraPro depth camera for acquiring visible light and depth image sequences, and a TB-4117-3 / s thermal infrared camera for acquiring high-precision temperature field distribution.

[0029] During the cruise, such as Figure 2 As shown in section (a), to ensure the accuracy of subsequent depth information calculations and the reliability of the infrared thermography physical compensation model, a preset observation distance, preferably 0.4m, is maintained between the multimodal sensor gimbal of the inspection robot and the pigeon cage mesh. Under this distance constraint, the system synchronously acquires visible light image sequences inside the pigeon cage. Depth image sequence Infrared thermal image sequence It also records the current ambient temperature in conjunction with onboard environmental sensors. relative humidity and ammonia concentration .

[0030] The acquired multimodal image sequences are divided into video segments according to time windows. Long videos are edited into several video segments with a duration of m minutes, and strict alignment of visible light, depth and infrared data in timestamps is ensured.

[0031] (1) Constructing the target detection dataset D1: Select key frames from visible light images and depth images, label the bounding box of the pigeon head target, and label the corresponding pose key point information; record the spatial coordinate information of the bounding box and key points, thereby constructing the target detection dataset D1, which is used to train the convolutional neural network.

[0032] (2) Constructing the micro-movement behavior recognition dataset D2: Extract continuous action sequences from video clip samples, and label the video clips according to the spatiotemporal motion changes of the pigeon's head in continuous frames. Classify the behavior categories into preset behavior categories such as effective pecking, drinking, feather pecking, stillness, and standing still near the trough, thereby constructing the micro-movement behavior recognition dataset D2. The micro-movement behavior recognition dataset D2 is used to train the micro-movement augmented state space network (ME-SSM).

[0033] (3) Constructing the health decision dataset D3: Based on the diagnosis results of breeding experts on the health status of individual squabs, the health status of squabs is divided into healthy status, sub-healthy status and high-risk status of disease, and the healthy status is used as the labeling result; at the same time, the corresponding core body temperature estimate, behavior classification results, behavior duration characteristics and environmental parameter information are recorded to construct the health decision dataset D3, which is used to train the random forest health assessment model.

[0034] Furthermore, the method for acquiring multimodal data frames is as follows: Multimodal video sequences include: visible light images, depth images, and infrared thermal images; Obtain the camera calibration parameters of the corresponding acquisition device for the multimodal video sequence; The depth image is reprojected onto the coordinate system of the visible light image based on the camera calibration parameters to obtain an aligned depth map; The homography matrix is ​​obtained based on the feature point matching relationship between visible light images and infrared thermal images; Geometric correction of infrared thermal images based on homography matrix yields aligned infrared thermal images; Based on the synchronization timestamps of the data collected by each acquisition device, the visible light image, the aligned depth map, and the aligned infrared thermal image are synchronized in time and matched at the frame level. The visible light image, the aligned depth map, and the aligned infrared thermal image that are aligned in both time and space are fused to construct a multimodal data frame under a unified coordinate system and a unified time reference.

[0035] Furthermore, the visible light camera, depth camera, and infrared thermal imager are jointly calibrated through a multimodal perception module to obtain the internal parameters and relative positional relationships of each camera, and spatial registration between multimodal images is achieved through geometric projection and feature matching methods.

[0036] Furthermore, the camera calibration parameters include: the visible light camera intrinsic parameter matrix. Depth camera intrinsic parameter matrix And the extrinsic parameter relationship between the two cameras, including the rotation matrix. Translation vector .

[0037] Furthermore, based on the calibration parameters, the reprojection geometric transformation relationship from the depth image to the visible light image coordinate system is established: ; in, p c Represents the pixel coordinates of the projected visible light image. This represents the inverse matrix of the depth camera intrinsic parameter matrix. p d Represents the pixel coordinates on the depth image. Z d This represents the pixel coordinates in the depth image. p d The actual depth value of the target point. and Let represent the rotation transformation matrix and translation transformation vector from the depth camera to the visible light camera, respectively; By performing the above reprojection calculation on each valid pixel in the depth image, a depth map aligned with the visible light image space is generated, i.e., an aligned depth map.

[0038] Furthermore, to achieve spatial alignment between the infrared thermal image and the visible light image, corresponding feature points are first extracted from the two images and a feature point matching relationship is established. After removing mismatched points, a homography matrix describing the mapping relationship between the two images is calculated. .

[0039] Pixel coordinates in an infrared thermal image With visible light image pixel coordinates p c The perspective mapping relationship between them can be represented as: p c = · ; Using homography matrix By performing a perspective transformation on the infrared thermal image, an infrared thermal image spatially aligned with the visible light image is obtained, i.e., an aligned infrared thermal image.

[0040] S2 obtains multi-dimensional spatial features of pigeon heads based on multimodal data frames.

[0041] Furthermore, the method for obtaining multidimensional spatial features is as follows: Based on the visible light image input to the convolutional neural network, the bounding box parameters of each pigeon's head are extracted; The bounding box region of the corresponding pigeon's head is determined based on the bounding box parameters. Extract the coordinates of key points on the pigeon's head based on the bounding box region; Map the bounding box region onto the aligned depth map, and extract the effective mean depth of the bounding box covered area as the physical distance of the pigeon's head relative to the acquisition device. Based on camera calibration parameters, aligned depth map and key point coordinates, the three-dimensional attitude deflection angle of the pigeon's head is obtained; Multidimensional spatial features are obtained by fusing bounding box parameters, key point coordinates, physical distance, and 3D attitude deflection angle.

[0042] Furthermore, the bounding box parameters of the pigeon's head B : ; in, Indicates the coordinates of the center point of the bounding box; These represent the width and height of the bounding box, respectively.

[0043] Key coordinates of a pigeon's head include the coordinates of the beak tip. Left eye position coordinates and the coordinates of the right eye position The above key points reflect the geometric structure of the pigeon's head, where the position of the eyes can represent the left-right direction of the head, while the position of the beak tip is used to represent the orientation of the head.

[0044] By combining bounding boxes and keypoints, spatial positioning information of the pigeon's head was constructed, providing a foundation for subsequent spatial parameter calculations.

[0045] Furthermore, the bounding box region is mapped to a depth map aligned with the visible light image space, and the set of valid depth values ​​within that region is extracted: ; in, Represents the depth image of the th n The depth value of each pixel. n Indicates the number of valid depth pixels within the bounding box region; The average statistical depth value was used as the physical distance between the pigeon's head and the data collection device. d : ; The above calculations can obtain the distance information of the pigeon's head in three-dimensional space, thus providing the necessary spatial parameters for the subsequent body temperature compensation model.

[0046] Furthermore, the method for obtaining the three-dimensional attitude deflection angle is as follows: Based on the aligned depth map and the camera intrinsic parameter matrix in the camera calibration parameters, a 3D back projection is performed on the key point coordinates to obtain the corresponding 3D spatial coordinates of the beak tip position. 3D spatial coordinates of the left eye position and the three-dimensional spatial coordinates of the right eye position ; Construct the direction vector of the eye-binding line based on the three-dimensional spatial coordinates of the left and right eye positions. Used to represent the left-right direction of a squab's head: ; Based on the three-dimensional spatial coordinates of the beak tip position, the three-dimensional spatial coordinates of the left eye position, and the three-dimensional spatial coordinates of the right eye position, a sagittal direction vector from the beak tip to the center of both eyes is constructed. Used to indicate the forward orientation of a squab's head: ; Based on the direction vector of the eye connection line and sagittal direction vector Obtain the spatial normal vector of the pigeon's head. : ; in, This represents the spatial normal vector that the pigeon's head is facing. Represents the space normal vector Components in a three-dimensional coordinate system; Based on space normal vector relative to the optical axis direction vector of the acquisition device The angle was calculated to obtain the three-dimensional attitude deflection angle of the pigeon's head. : ; in, This represents the optical axis direction vector of the acquisition device. This represents the three-dimensional attitude deflection angle of the pigeon's head relative to the optical axis of the acquisition device. Through the above calculations, the attitude change information of the pigeon's head in three-dimensional space can be obtained.

[0047] Furthermore, the multidimensional spatial characteristics of individual pigeons : .

[0048] S3 uses multi-dimensional spatial features to perform identity association matching on individual pigeons in consecutive frames and generates time-series trajectories.

[0049] Furthermore, the generation of time-series trajectories specifically includes: The multidimensional spatial matching distance is based on the matching distance between the multidimensional spatial features corresponding to any two different pigeons in adjacent frames. The association cost matrix between all pigeons in the current frame and the previous frame is constructed based on the multidimensional spatial matching distance as the association cost. The Hungarian algorithm is used to perform global optimal matching based on the correlation cost matrix; Determine whether the association cost of a successfully matched target pair is less than a preset association threshold; If so, the corresponding target is determined to belong to the same individual pigeon, and its unique identity identifier (ID) is inherited. Otherwise, it is identified as a new target and a new identifier is assigned; By concatenating the multidimensional spatial features of individual pigeons with the same identity in consecutive video frames in chronological order, a temporal trajectory of the corresponding individual pigeon is generated.

[0050] Furthermore, for the current frame t The first in i Each detected target is compared to the previous frame. t -1 of the j For each detection target, the multidimensional spatial matching distance is calculated by examining the matching distance between their multidimensional spatial features. : ; in, This represents the multidimensional spatial matching distance between the i-th detected target in the current frame and the j-th detected target in the previous frame, and is used to characterize the association cost; and These represent the center pixel coordinates of the bounding box of the corresponding target in the current frame and the previous frame, respectively; and These represent the physical distances of the corresponding targets in the current frame and the previous frame, respectively. and These represent the three-dimensional attitude deflection angles of the corresponding target in the current frame and the previous frame, respectively; Represents norm operations; and These are the distance weighting coefficient and the pose weighting coefficient, respectively, used to balance the numerical differences between different features.

[0051] S4 constructs a temporal observation feature sequence based on temporal trajectory and corresponding multidimensional spatial features, and extracts the behavioral classification results and behavioral duration features of the corresponding pigeon individuals as behavioral data.

[0052] Furthermore, the method for constructing the time-series observation feature sequence is as follows: Based on the time-series trajectory, the multidimensional spatial feature changes of the same squab individual between adjacent time frames are obtained as micro-motion residuals; By fusing the micro-motion residuals with the corresponding multi-dimensional spatial features, a time-series observation vector characterizing the dynamic behavior of squabs is constructed. Based on the temporal observation vectors in continuous time frames arranged in chronological order, a temporal observation feature sequence of individual pigeons is constructed.

[0053] Furthermore, the micro-motion residual Specifically: ; in, Indicates time The multi-dimensional spatial features of the pigeon's head This represents the multi-dimensional spatial features of the pigeon's head in the previous time frame, and the micro-motion residual can characterize the subtle motion changes of the pigeon's head in continuous video frames.

[0054] Furthermore, the original multidimensional spatial features With micro-motion residual By splicing and fusing feature dimensions, a complete temporal observation vector representing the dynamic behavior of squabs is constructed: ; in, Indicates time The time-series observation vector, when multidimensional spatial features When it is 12-dimensional, the time series observation vector It is 24-dimensional.

[0055] observation vectors in continuous time frames Arranged in chronological order, that is, constructing a temporal observation feature sequence of individual pigeons.

[0056] Furthermore, the method for acquiring behavioral data is as follows: By inputting the time-series observation feature sequence into the micro-motion enhanced state space network, the behavior of individual pigeons is modeled in time series to obtain historical state information vectors. When a target is occluded, the target location is predicted based on the historical state information vector to maintain the continuity of the target identity sequence. At the same time, the historical state information vector is fused with the temporal observation vector to obtain behavioral features. Based on behavioral characteristics, a probability distribution of individual pigeons in each behavioral category is generated, and the behavioral category with the highest probability is selected as the behavioral classification result according to the probability distribution. The number of consecutive frames of behavior classification results within a continuous time window is counted, and the behavior duration feature is obtained by combining the sampling frame rate. Behavioral classification results and behavioral duration characteristics together constitute behavioral data.

[0057] Furthermore, the observation vector Inputting a Micro-Motion Enhanced State-Space Network (ME-SSM) to perform temporal modeling of individual pigeon behavior, the historical state information update process is represented as follows: ; in, h t Indicates time The historical state information vector, h t-1 Indicates time -1 historical state information vector A Represents the state transition matrix. B This represents the input mapping matrix. Through the recursive update mechanism described above, the model can maintain historical behavioral information over continuous time series, thereby achieving long-term time-series modeling of the behavioral changes of individual pigeons.

[0058] To further enhance the model's ability to represent high-frequency minute movements, a micro-motion feedforward gain mechanism is introduced in the model output stage. The output process is represented as follows: ; in, Indicates behavioral characteristics, Represents the state observation matrix, This represents the micro-motion feedforward gain matrix. By introducing the feedforward gain matrix... It can fuse micro-motion residuals with state features in the output stage, thereby reducing the high-frequency information decay in the state recursion process and improving the model's ability to express fine-grained action changes. In particular, when individual pigeons experience grid occlusion or short-term detection loss, the network can use the historical state information vector at the current moment to infer the target position and state, thereby achieving identity sequence preservation and trajectory continuity.

[0059] Furthermore, the obtained behavioral characteristics Mapping to the behavior category space to obtain the individual pigeon at time [time]. Behavioral probability distribution: ; in, This represents the probability distribution of individual pigeons across various behavioral categories. Represents the behavior classification weight matrix. Indicates category bias; Based on behavioral probability distribution The category with the highest probability value is selected as the behavior classification result of the pigeon individual at the current moment, and this behavior category is denoted as... . Furthermore, the behavioral categories include: effective pecking, drinking, feather pecking, stillness, and standing still near the crate.

[0060] Furthermore, the number of consecutive frames for the same behavior category within a continuous time window is counted, and the duration of the corresponding behavior is calculated in conjunction with the system's sampling frame rate. This duration feature of the behavior is denoted as... This is used to characterize the duration of an individual pigeon's behavioral state within the current time period.

[0061] S5 obtains the infrared observation temperature of the pigeon's head based on multimodal data frames, and performs temperature compensation to obtain the pigeon's core body temperature.

[0062] Furthermore, the method for obtaining the core body temperature of squabs is as follows: Infrared observation temperature of pigeon head extracted from aligned infrared thermograms in multimodal data frames; A segmented thermal radiation compensation model is constructed based on a preset ranging boundary threshold, and distance compensation is performed on the infrared observation temperature according to the physical distance to obtain the distance-compensated temperature. The angular radiation attenuation factor is obtained based on the preset infrared emissivity coefficient of pigeon feathers and the three-dimensional attitude deflection angle. The core body temperature of pigeons was obtained based on distance-compensated temperature and angular radiation attenuation factor.

[0063] Furthermore, based on the preset ranging boundary threshold Construct a segmented thermal radiation compensation model: When the distance between the pigeon's head and the sensor is less than the ranging boundary threshold... At that time, a polynomial compensation model was used to analyze the infrared temperature. Perform correction: ; When the distance between the pigeon's head and the sensor is greater than or equal to the ranging threshold At that time, a logarithmic decay compensation model is used for correction: ; in, This indicates the distance-compensated temperature after distance compensation. , as well as To compensate for model parameters; This represents the physical distance between the pigeon's head and the infrared sensor. By employing a segmented compensation model, the temperature fitting accuracy can be improved when measuring temperature at close range, and the influence of environmental absorption and radiation attenuation on the temperature measurement results can be compensated when measuring at long distances.

[0064] Furthermore, after distance compensation, to further reduce the impact of pigeon head posture changes on infrared radiation measurement results, a three-dimensional posture deflection angle was used as the basis for... Combining the infrared emissivity coefficient of pigeon feathers The angular radiation attenuation factor is calculated based on the corrected Lambert cosine law. : ; Based on angular radiation attenuation factor Distance-compensated temperature Posture correction was performed to obtain the squab's core body temperature. T c : ; By combining the above-mentioned distance compensation and posture correction calculations, the influence of changes in measurement distance and head posture deflection on infrared thermometry results can be effectively reduced, thereby obtaining a temperature estimate that is closer to the true core body temperature of pigeons.

[0065] S6 obtains health status characteristics by fusing behavioral data, squab core body temperature, and environmental parameters, and acquires squab health assessment and grading results.

[0066] Furthermore, obtain the health assessment and grading results of the squabs, specifically including: Based on the synchronization timestamp, the core body temperature, behavior classification results, behavior duration features and environmental parameters of pigeons are time-aligned and resampled, and the health status feature vector is obtained by splicing the features of various multi-source data after preprocessing. The health status feature vector is input into the random forest health assessment model to classify and assess the health status of individual pigeons, and the corresponding health assessment grade results of individual pigeons are obtained. The random forest health assessment model is trained based on historical physiological data, behavioral category data, and veterinary clinical diagnostic sample data. It establishes a mapping relationship between the physiological state, behavioral patterns, and health status of pigeons through an ensemble structure of multiple decision trees.

[0067] Furthermore, individual pigeons at any given time t Health status feature vector : ; in, Indicates the individual squab at any given time t The core body temperature of the squab; c t Indicates the behavior classification result; l t Indicates the duration of the behavior; e t This represents an environmental parameter vector, including environmental information such as ambient temperature, relative humidity, and ammonia concentration.

[0068] Example 2 The following are the test results of pigeon fine behavior identification, body temperature and health using the test set of the embodiments of the present invention: In this invention, the duration of the preprocessed long video is set. Time window width sliding step size Several multimodal video clip samples were constructed using sliding window segmentation technology. To verify the model's ability to distinguish highly similar behaviors, the samples covered five typical behaviors of squabs, specifically effective pecking, standing still near the trough, drinking, feather pecking, and stillness. During data annotation, the true bounding box of the squab's head, physical distance, core body temperature, and corresponding behavior classification labels were recorded simultaneously for each sample, thus constructing a multimodal health assessment validation dataset.

[0069] During the test, the video was viewed from the front. The second is defined as the input step size. Within this time period, the system synchronously extracts multimodal image frames and performs the following operations: (1) The bounding box and key pixel coordinates of the pigeon's head are extracted using a convolutional neural network, and the physical distance of the pigeon's head is obtained by three-dimensional back projection based on the spatially aligned depth map. and three-dimensional attitude deflection angle ; (2) Combine the above spatial features to construct a multidimensional spatial matching distance to achieve stable tracking of the target identity, and extract the multidimensional spatial feature change of the same identity identifier between adjacent frames as micro-motion residual, and splice and fuse it with the multidimensional spatial features; (3) Input the temporal observation feature sequence containing inter-frame micro-motion residuals into the micro-motion enhanced state space network (ME-SSM) for long-term temporal modeling and anti-occlusion inference, and output the fine-grained behavior classification results and behavior duration characteristics of individual pigeons; (4) Compensate for the infrared observation temperature by introducing physical distance. and three-dimensional attitude deflection angle Dynamic correction of infrared temperature observations for spatial scale and three-dimensional attitude was performed to obtain an estimated value of the core body temperature of pigeons. (5) The behavior classification results, behavior duration characteristics, core body temperature estimates and environmental parameters are time-aligned and fused to construct a health status feature vector. Based on the health status feature vector, a pre-trained random forest health assessment model is input for classification assessment, and the health assessment grade of individual pigeons is output.

[0070] To verify the effectiveness of the pigeon health assessment method based on multimodal data fusion provided in this embodiment of the invention, this embodiment uses mean precision (mAP), F1 score (F1-Score), and mean absolute error (MAE) as performance evaluation indicators. The specific evaluation method includes the following: (1) Target detection accuracy evaluation: The mean accuracy (mAP) is used to measure the detection accuracy of convolutional neural networks for the bounding box and key points of pigeon heads in a high-density cage occlusion environment.

[0071] (2) Fine-grained behavior classification evaluation: The precision and recall of the F1 score model are combined to evaluate the accuracy of the behavior classification results of the micro-motion enhanced state space network (ME-SSM) in distinguishing between effective pecking and near-groove standing behavior.

[0072] (3) Assessment of body temperature estimation error: Mean absolute error (MAE) is used to measure the average absolute error between the estimated core body temperature of pigeons calculated based on dual dynamic compensation of distance and angle and the actual core body temperature obtained by veterinary clinical measurement.

[0073] (4) Detection matching and statistical benchmark: During the evaluation process, for each frame of multimodal data, based on the preset intersection-union ratio (IoU) threshold, the bounding boxes and classification results predicted by the system are matched with the corresponding labeled ground truth values ​​in the dataset, and statistical evaluation is performed accordingly.

[0074] After testing, the method of the present invention can achieve robust and efficient multimodal assessment and classification performance even in environments with severe shading of cage mesh, feather reflection, and dense activity, which is significantly better than existing single-modal poultry monitoring methods.

[0075] In summary, this invention provides an effective method for assessing the health of squabs in high-density cage environments. This invention enables accurate positioning of the squab's head, dynamic compensation estimation of core body temperature, and fine-grained behavior recognition based on time-series observation features. Furthermore, it integrates a random forest health assessment model to comprehensively evaluate classification results, behavioral duration characteristics, body temperature characteristics, and environmental parameters, thereby achieving automated monitoring of individual squab health status. Compared to traditional manual inspection methods, this invention improves the automation and monitoring efficiency of squab health assessment, reduces infrared thermometry errors and similar behavior recognition errors, and is suitable for assessing individual health status in large-scale poultry farming environments.

[0076] Example 3 like Figure 3As shown, based on the same inventive concept, this embodiment of the invention also provides a pigeon health assessment system based on multimodal data fusion, including: a data acquisition and processing module, a head feature extraction module, a time-series trajectory generation module, a behavioral data acquisition module, a core body temperature acquisition module, and an assessment result output module; The data acquisition and processing module is used to acquire the multimodal video sequence of the target pigeon cage and perform preprocessing to obtain multimodal data frames; The head feature extraction module is used to obtain multi-dimensional spatial features of pigeon heads based on multimodal data frames; The temporal trajectory generation module is used to perform identity association matching on individual pigeons in continuous frames based on multi-dimensional spatial features to generate temporal trajectories. The behavior data acquisition module is used to construct a time-series observation feature sequence based on the time-series trajectory and corresponding multi-dimensional spatial features, and extract the behavior classification results and behavior duration features of the corresponding pigeon individuals as behavior data; The core body temperature acquisition module is used to acquire the infrared observation temperature of the pigeon's head based on multimodal data frames, and perform temperature compensation to obtain the pigeon's core body temperature. The assessment results output module is used to obtain health status characteristics based on behavioral data, squab core body temperature and environmental parameters, and to obtain the squab health assessment grading results.

[0077] Furthermore, in this embodiment, the functional implementation methods of each functional module correspond one-to-one with the methods described above, and will not be repeated here.

[0078] Example 4 Based on the same inventive concept, the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores instructions, characterized in that the instructions are loaded and executed by the processor to implement a pigeon health assessment method based on multimodal data fusion as in Embodiment 1.

[0079] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When the processor executes the program stored in the memory, it is able to implement a pigeon health assessment method based on multimodal data fusion, as shown in Example 1.

[0080] The electronic device may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute a pigeon health assessment method based on multimodal data fusion as described in Embodiment 1.

[0081] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0083] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for assessing the health of squabs based on multimodal data fusion, characterized in that, include: The multimodal video sequence of the target pigeon cage is acquired and preprocessed to obtain multimodal data frames; The multidimensional spatial features of the pigeon's head are obtained based on the multimodal data frames. Based on the multidimensional spatial features, identity association matching is performed on individual pigeons in consecutive frames to generate time-series trajectories; Based on the time-series trajectory and the corresponding multidimensional spatial features, a time-series observation feature sequence is constructed, and the behavioral classification results and behavioral duration features of the corresponding pigeon individuals are extracted as behavioral data. The infrared observation temperature of the pigeon's head is obtained based on the multimodal data frame, and temperature compensation is performed to obtain the pigeon's core body temperature. Based on the behavioral data, the squab's core body temperature, and environmental parameters, health status characteristics are obtained, and the squab health assessment and grading results are acquired.

2. The method for assessing pigeon health based on multimodal data fusion as described in claim 1, characterized in that, The method for acquiring multimodal data frames is as follows: The multimodal video sequence includes: visible light images, depth images, and infrared thermal images; Obtain the camera calibration parameters of the corresponding acquisition device for the multimodal video sequence; Based on the camera calibration parameters, the depth image is reprojected onto the coordinate system of the visible light image to obtain an aligned depth map; The homography matrix is ​​obtained based on the feature point matching relationship between the visible light image and the infrared thermal image; Geometric correction is performed on the infrared thermal image based on the homography matrix to obtain an aligned infrared thermal image; Based on the synchronization timestamps of the data collected by each acquisition device, the visible light image, the aligned depth map, and the aligned infrared thermal image are synchronized in time and matched at the frame level. The visible light image, the aligned depth map, and the aligned infrared thermal image, which are aligned in both time and space, are fused to construct the multimodal data frame in a unified coordinate system.

3. The method for assessing pigeon health based on multimodal data fusion as described in claim 2, characterized in that, The method for obtaining multidimensional spatial features is as follows: Based on the visible light image input to the convolutional neural network, the bounding box parameters of each pigeon's head are extracted; The bounding box region of the corresponding pigeon's head is determined based on the bounding box parameters. Based on the bounding box region, extract the coordinates of key points on the pigeon's head; The bounding box region is mapped onto the alignment depth map, and the effective mean depth of the bounding box covered area is extracted as the physical distance of the pigeon's head relative to the acquisition device. Based on the camera calibration parameters, the alignment depth map, and the key point coordinates, the three-dimensional attitude deflection angle of the pigeon's head is obtained; The multidimensional spatial features are obtained by fusing the bounding box parameters, the key point coordinates, the physical distance, and the three-dimensional attitude deflection angle.

4. The method for assessing pigeon health based on multimodal data fusion as described in claim 3, characterized in that, The method for obtaining the three-dimensional attitude deflection angle is as follows: Based on the alignment depth map and the camera intrinsic parameter matrix in the camera calibration parameters, the coordinates of the key points are back-projected in three dimensions to obtain the three-dimensional spatial coordinates of the beak tip position, the three-dimensional spatial coordinates of the left eye position and the three-dimensional spatial coordinates of the right eye position. Construct a direction vector for the line connecting the two eyes based on the three-dimensional spatial coordinates of the left eye position and the three-dimensional spatial coordinates of the right eye position; Based on the three-dimensional spatial coordinates of the beak tip position, the three-dimensional spatial coordinates of the left eye position, and the three-dimensional spatial coordinates of the right eye position, a sagittal direction vector from the beak tip to the center of both eyes is constructed; The spatial normal vector of the pigeon's head is obtained based on the direction vector of the eye-to-eye line and the direction vector of the sagittal plane. The angle between the spatial normal vector and the optical axis direction vector of the acquisition device is calculated to obtain the three-dimensional attitude deflection angle of the pigeon's head.

5. The method for assessing pigeon health based on multimodal data fusion as described in claim 4, characterized in that, Generating time-series trajectories specifically includes: The multidimensional spatial matching distance is based on the matching distance between the multidimensional spatial features corresponding to any two different pigeons in adjacent frames. Based on the multidimensional spatial matching distance as the association cost, construct the association cost matrix between all pigeon individuals in the current frame and the previous frame; Based on the aforementioned correlation cost matrix, the Hungarian algorithm is used for global optimal matching; Determine whether the association cost corresponding to the successfully matched target pair is less than a preset association threshold; If so, the corresponding target is determined to belong to the same individual pigeon, and its unique identity is inherited; Otherwise, it is identified as a new target and a new identifier is assigned; The multidimensional spatial features corresponding to individual pigeons with the same identity in consecutive video frames are concatenated in chronological order to generate the temporal trajectory of the corresponding individual pigeon.

6. The method for assessing pigeon health based on multimodal data fusion as described in claim 5, characterized in that, The method for constructing time-series observation feature sequences is as follows: Based on the time-series trajectory, the multidimensional spatial feature changes of the same squab individual between adjacent time frames are obtained as micro-motion residuals. Based on the fusion of the micro-motion residuals and the corresponding multi-dimensional spatial features, a time-series observation vector characterizing the dynamic behavior of squabs is constructed. The temporal observation feature sequence of an individual pigeon is constructed by arranging the temporal observation vectors in continuous time frames in chronological order.

7. The method for assessing pigeon health based on multimodal data fusion as described in claim 6, characterized in that, The method for acquiring the behavioral data is as follows: Based on the input of the time-series observation feature sequence into the micro-motion enhanced state space network, the time-series behavior of individual pigeons is modeled to obtain a historical state information vector. When the target is occluded, the target position is predicted based on the historical state information vector to maintain the continuity of the target identity sequence. At the same time, the historical state information vector is fused with the temporal observation vector to obtain the behavioral features. Based on the behavioral characteristics, a probability distribution of individual pigeons in each behavioral category is generated, and the behavioral category with the highest probability is selected as the behavioral classification result according to the probability distribution. The number of consecutive frames of the behavior classification results within a continuous time window is counted, and the duration feature of the behavior is obtained by combining the sampling frame rate. The behavior classification results and the behavior duration features together constitute the behavior data.

8. The method for assessing pigeon health based on multimodal data fusion as described in claim 7, characterized in that, The method for obtaining the core body temperature of squabs is as follows: The infrared observation temperature of the pigeon's head is extracted based on the aligned infrared thermogram in the multimodal data frame; A segmented thermal radiation compensation model is constructed based on a preset ranging boundary threshold, and the infrared observation temperature is compensated for distance according to the physical distance to obtain the distance-compensated temperature. The angular radiation attenuation factor is obtained based on the preset infrared emissivity coefficient of pigeon feathers and the three-dimensional attitude deflection angle. The core body temperature of the pigeon is obtained based on the distance-compensated temperature and the angular radiation attenuation factor.

9. The method for assessing pigeon health based on multimodal data fusion as described in claim 7, characterized in that, Obtain the health assessment and grading results for squabs, specifically including: Based on the synchronization timestamp, the core body temperature of the squab, the behavior classification results, the duration of the behavior, and the environmental parameters are time-aligned and resampled, and the health status feature vector is obtained by splicing the features of various preprocessed multi-source data. The health status feature vector is input into the random forest health assessment model to classify and assess the health status of individual pigeons, and the corresponding health assessment grade result of the individual pigeon is obtained. The random forest health assessment model is trained based on historical physiological data, behavioral category data, and veterinary clinical diagnostic sample data. It establishes a mapping relationship between the physiological state, behavioral patterns, and health status of pigeons through an integrated structure of multiple decision trees.

10. A pigeon health assessment system based on multimodal data fusion, used to execute the pigeon health assessment method based on multimodal data fusion as described in any one of claims 1-9, characterized in that, include: The system includes a data acquisition and processing module, a head feature extraction module, a time-series trajectory generation module, a behavioral data acquisition module, a core body temperature acquisition module, and an evaluation result output module. The data acquisition and processing module is used to acquire the multimodal video sequence of the target pigeon cage and preprocess it to obtain multimodal data frames; The head feature extraction module is used to obtain multi-dimensional spatial features of the pigeon's head based on the multimodal data frame. The time-series trajectory generation module is used to perform identity association matching on individual pigeons in continuous frames based on the multi-dimensional spatial features to generate time-series trajectories. The behavior data acquisition module is used to construct a time-series observation feature sequence based on the time-series trajectory and the corresponding multi-dimensional spatial features, and extract the behavior classification results and behavior duration features of the corresponding pigeon individuals as behavior data. The core body temperature acquisition module is used to acquire the infrared observation temperature of the pigeon's head based on the multimodal data frame, and perform temperature compensation to obtain the pigeon's core body temperature. The assessment result output module is used to obtain health status characteristics based on the fusion of the behavioral data, the squab's core body temperature, and environmental parameters, and to obtain the squab's health assessment grading results.

Images