Poultry behavior abnormity real-time monitoring system based on multi-modal image fusion

Abstract

The invention discloses a poultry behavior abnormity real-time monitoring system based on multi-modal image fusion, particularly relates to the technical field of intelligent breeding behavior recognition, and is used for solving the problem of poor behavior monitoring accuracy under feather shielding. The method comprises the following steps: firstly, through combined perception of a visible light image and an infrared image, extracting a claw track interruption point and an anus temperature gradient direction, and realizing analysis of a motion state of a sheltered area; then, in combination with the heat conduction delay characteristic and the group movement direction, the flexion and extension angle of the covered leg joint is inverted, and a complete gait sequence is generated; thirdly, multi-source features such as gaits, temperature differences and body postures are fused, and a dynamic deviation model of the individuals relative to the mass center of the group is constructed; and finally, generating a stress behavior threshold curve according to the ground temperature and the ammonia gas concentration, outputting an abnormal behavior type and confidence, and realizing intelligent distinguishing of mechanical obstacles and adaptive behaviors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent breeding behavior recognition, in particular to a poultry behavior abnormality real-time monitoring system based on multi-modal image fusion. BACKGROUND

[0002] With the continuous improvement of the intelligent level of livestock and poultry breeding, real-time identification and automatic early warning of individual behavior states of poultry based on computer vision technology have become an important part of digital and fine management of the breeding process. In recent years, multi-modal image perception methods have gradually been popularized in breeding scenes. Infrared thermal imaging, visible light video and depth vision systems are widely used for behavior observation, physical sign judgment and health risk warning. Identifying and modeling the gait, posture and activity trajectory of poultry through image sequences helps to timely detect movement disorders, abnormal stress and early signs of disease, and improve breeding efficiency and animal welfare level.

[0003] However, the existing technology has a series of key problems in dealing with poultry behavior monitoring tasks, which affects the application accuracy and robustness of the system. First, a single image modality cannot adapt to the frequent occurrence of occlusion, shielding and image distortion in the actual breeding environment, especially in the dense feather area, the target boundary is blurred, which leads to the breaking or loss of behavior elements, making it difficult to reconstruct the complete action sequence. Second, although the infrared thermal image can assist in locating the thermal conduction characteristics of the occlusion area, most existing methods are based on static temperature field to determine the state, lacking the ability to cooperatively model the time sequence image behavior events, making it difficult to realize dynamic gait restoration. Third, behavior recognition models are mostly based on local image features or two-dimensional plane action judgment, and cannot effectively integrate the temporal consistency, three-dimensional spatial correlation and environmental context among multi-source images, limiting the discrimination accuracy of the system for abnormal behavior patterns. In addition, under the influence of various environmental factors, the existing model cannot flexibly adjust the classification criteria or behavior threshold, which easily leads to fuzzy classification boundaries, high misjudgment rate and other problems, affecting accurate differentiation of different types of abnormal behaviors.

[0004] In summary, the existing poultry behavior monitoring system has substantial processing bottlenecks in image fusion capability, temporal restoration under occlusion conditions, three-dimensional behavior modeling and environmental correlation anomaly discrimination, and an image processing and recognition system based on multi-modal image fusion is urgently needed to robustly identify occluded targets and dynamically analyze behavior changes. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a poultry behavior abnormality real-time monitoring system based on multi-modal image fusion, which solves the problems of the above background technology.

[0006] In order to achieve the above object, the present application is realized by the following technical scheme: the poultry behavior abnormality real-time monitoring system based on multi-modal image fusion, comprising the following modules: occluded target perception module, behavior inversion module, fusion analysis module, behavior decision module; the occluded target perception module is used for positioning the poultry target of the feather high-occluded area through the visible light image, extracting the breakpoint in the claw swing trajectory and the displacement vector of the head exposure area, and simultaneously analyzing the annular distribution gradient direction of the anal temperature according to the infrared image; the behavior inversion module is used for inferring the joint flexion angle of the covered leg based on the heat conduction delay characteristic of the anal temperature gradient direction, combining the spatial consistency constraint of the claw trajectory breakpoint and the moving direction of the adjacent poultry group, and generating the repaired gait sequence; the fusion analysis module matches the repaired gait sequence with the body tilt angle change in the visible light image on the time axis, synchronously fuses the left and right foot temperature difference characteristics in the infrared image, and calculates the dynamic deviation coefficient of the individual motion trajectory deviating from the group centroid according to the three-dimensional position distribution of the chicken group; the behavior decision module establishes the environmental stress behavior threshold curve according to the ground temperature and the ammonia concentration, distinguishes the mechanical movement disorder and the environmental adaptability behavior through the coupling relationship of the support phase length, the body tilt angle and the motion trajectory dynamic deviation value in the gait sequence, and outputs the abnormal behavior type and the confidence.

[0007] Further, the processing process of the occluded target perception module comprises the following contents: frame-by-frame scanning of the visible light video stream, identification of the claw joint area in the feather gap through a multi-scale texture matching algorithm, analysis of the gray scale distribution of the visible light image, and output of the feather covering density level; the position change of each joint is tracked, the claw swing trajectory is constructed, and the displacement vector between adjacent frames is calculated, and when the displacement vector exceeds the preset continuity threshold, it is marked as a breakpoint; the three-dimensional space coordinates of the head exposure area are calculated through the perspective of the binocular camera, the pitch angle and the yaw angle of the head are calculated according to the three-dimensional coordinate change of the continuous frames, and the displacement vector is calculated, and the displacement direction and amplitude are quantified in combination with the angle change rate.

[0008] Further, the processing process of the occluded target perception module further comprises the following contents: dynamically adjusting the segmentation threshold according to the mapping relationship between the predefined environmental temperature and the infrared image segmentation threshold, binarizing and segmenting the infrared image through the adjusted threshold, filling the holes in combination with the morphological closing operation, and obtaining the connected region of the anal area; the temperature gradient of the outline edge of the anal area is calculated, the edge point with the maximum gradient value is extracted and connected into a line, and the direction angle of the line is calculated; according to the pre-established mapping relationship between the gradient direction angle and the leg muscle contraction, the current gradient direction angle is converted into the corresponding leg muscle contraction state, and the muscle activation intensity code is output.

[0009] Further, the behavior inversion module includes the following: determining the heat conduction path of the covered leg according to the muscle group activation intensity coding, and measuring the delay time of heat conduction; loading the corresponding heat conduction rate correction coefficient according to the feather covering density level, and calculating the actual heat conduction delay time by applying the conduction rate correction coefficient; calculating the joint flexion angle according to the pre-marked delay time and joint flexion angle relationship curve.

[0010] Further, the behavior inversion module further includes the following: identifying the start and end frame positions of the claw trajectory breakpoint, obtaining the moving direction vector of the adjacent poultry within a radius setting range centered on the target poultry, and fusing the joint flexion angle and the neighbor moving direction vector to generate a spatial constraint direction; generating a repaired trajectory for the interruption interval using cubic spline interpolation according to the claw trajectory trend before the breakpoint and the spatial consistency constraint direction, connecting the repaired trajectory with the original trajectory to form a complete gait sequence.

[0011] Further, the fusion analysis module includes the following: extracting the foot landing time stamp in the repaired gait sequence as the reference time axis, resampling the spine curvature angle change sequence in the visible light image to the time axis, and eliminating the collection time error of the gait sequence and the body tilt angle sequence by the dynamic time warping algorithm; segmenting the thermal radiation profile of the left and right foot bottom regions in the infrared image, calculating the average temperature difference of the foot center region to generate a temperature difference sequence, and phase delay compensation aligning the temperature difference sequence with the gait support phase start time point.

[0012] Further, the fusion analysis module further includes the following: reconstructing a three-dimensional spatial distribution model of the chicken group through depth camera point cloud data, and estimating the body size index of each poultry based on the visible light image; calculating the weighted group centroid coordinates with the body size index square as the weight, sampling the individual position coordinates at equal intervals within the gait cycle, and calculating the Euclidean distance from each sampling point to the weighted centroid trajectory in real time, and taking the ratio of the standard deviation to the mean of the Euclidean distance sequence as the dynamic deviation coefficient.

[0013] Further, the root behavior decision module includes the following: establishing a two-dimensional parameter coordinate system of ground temperature and ammonia concentration, laying data collection grid points in the parameter space, and recording the support phase duration and body tilt angle range of normal poultry at each grid point; fitting the support phase duration threshold surface by applying the radial basis function kernel, establishing the body tilt angle tolerance boundary model by polynomial regression, and obtaining the current environmental parameter coordinate point in real time, and interpolating the stress behavior threshold value in the threshold surface and the boundary model.

[0014] Further, the root behavior decision module further comprises the following: normalizing the support phase duration according to the environmental stress behavior threshold curve, normalizing the body tilt angle according to the tolerance boundary, and implementing the density adaptive scaling of the motion trajectory dynamic deviation coefficient; constructing a three-dimensional feature vector containing the normalized support phase ratio, the normalized tilt angle and the scaled deviation coefficient; mapping the feature vector to the decision space through the pre-trained classifier, calculating the confidence according to the distance of the feature vector to the classification boundary, and determining the mechanical movement disorder when the support phase ratio continuously exceeds the threshold and the deviation coefficient is abnormal, and marking the environmental adaptive behavior when the tilt angle is temporarily over-standard but other features are normal.

[0015] The present application has the following advantages:

[0016] (1) The poultry behavior abnormality real-time monitoring system based on multi-modal image fusion effectively solves the problem of acquiring key posture information of poultry under high occlusion and complex motion state through the collaborative processing of the occlusion target perception module and the behavior inversion module. The system uses texture structure analysis and displacement vector extraction in visible light images, combines with temperature gradient features in infrared images, realizes accurate perception of target poultry in the feather occluded area and temperature field modeling. Further, the occluded leg joint flexion angle is inferred combined with the temperature conduction delay characteristics, and the interrupted gait trajectory is reconstructed continuously by fusing the group motion consistency constraint, which overcomes the problems of low action recovery accuracy and poor model stability in the image occluded area of traditional methods.

[0017] (2) The poultry behavior abnormality real-time monitoring system based on multi-modal image fusion realizes spatio-temporal coupling analysis and intelligent judgment of multi-dimensional heterogeneous information through the fusion analysis module and the behavior decision module. The system aligns the gait sequence and the posture angle change on the time axis, combines with the foot temperature difference and the spatial distribution data, and constructs the cross-modal behavior feature representation; and further introduces the dynamic deviation coefficient to describe the difference mode of individual and group behavior. In the behavior decision stage, the parameterized stress response model is established according to the ground temperature and ammonia concentration, and the coupling relationship among the gait support phase duration, the body tilt angle and the deviation coefficient is used to realize the accurate classification of mechanical movement disorder and environmental adaptive behavior, which improves the recognition ability and classification confidence of the system to early abnormal behavior, and solves the problems of insufficient generalization ability and response lag in multi-source data fusion analysis and abnormal type distinction of existing methods.

[0018] Of course, implementing any product of the present application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The flowchart of the poultry behavior abnormality real-time monitoring system based on multi-modal image fusion of the present application. DETAILED DESCRIPTION

[0020] The poultry behavior anomaly real-time monitoring system based on multi-modal image fusion provided by the embodiment of the present application solves the problems of unstable target posture extraction, low behavior anomaly detection precision and difficult type differentiation caused by visible light image occlusion, limited single modal feature expression ability and nonlinear behavior mode factors in actual breeding environment.

[0021] The scheme in the embodiment of the present application has the following general idea:

[0022] The system first uses the visible light image to perform texture analysis and target positioning through the occluded target perception module, extracts the claw movement breakpoint and head displacement vector, and simultaneously extracts the anus temperature annular gradient direction from the infrared image to assist in judging the motion state of the occluded part; then, the behavior inversion module reconstructs the gait sequence of the occluded area according to the heat conduction delay characteristic and the spatial consistency constraint to repair the discontinuous trajectory.

[0023] Next, the fusion analysis module aligns and fuses the multi-modal features such as gait, posture and temperature in the time dimension, and calculates the degree of individual deviation from the group behavior based on the three-dimensional distribution model of point cloud reconstruction; finally, the behavior decision module constructs a behavior threshold model in combination with the environmental parameters, discriminates the mechanical movement disorder and the environmental adaptive behavior through the coupling relationship of multi-dimensional features, and outputs the abnormal type and confidence. The system has the characteristics of strong multi-source information fusion, high anomaly recognition precision and strong adaptability to complex environment.

[0024] Please refer to Figure 1 The embodiment of the present application provides a technical scheme: a poultry behavior anomaly real-time monitoring system based on multi-modal image fusion, comprising the following modules: an occluded target perception module, a behavior inversion module, a fusion analysis module and a behavior decision module; the occluded target perception module is used to locate the poultry target in the high occlusion area of feathers through the visible light image, extract the claw swing trajectory breakpoint and head exposure area displacement vector, and simultaneously analyze the annular distribution gradient direction of the anus temperature according to the infrared image; the behavior inversion module is used to infer the joint flexion angle of the covered leg based on the heat conduction delay characteristic of the anus temperature gradient direction, generate the repaired gait sequence in combination with the spatial consistency constraint of the claw trajectory breakpoint and the moving direction of the adjacent poultry group; the fusion analysis module matches the repaired gait sequence with the body tilt angle change in the visible light image in the time axis, synchronously fuses the left and right foot temperature difference features in the infrared image, and calculates the dynamic deviation coefficient of the individual motion trajectory from the group centroid according to the three-dimensional position distribution of the chicken group; the behavior decision module establishes the environmental stress behavior threshold curve according to the ground temperature and ammonia concentration, discriminates the mechanical movement disorder and the environmental adaptive behavior through the coupling relationship of the support phase length, body tilt angle and motion trajectory dynamic deviation value in the gait sequence, and outputs the abnormal behavior type and confidence.

[0025] In this embodiment, the occlusion target perception module is mainly used to solve the problem of missing visible light image information caused by feather occlusion, group overlap and other factors in poultry behavior monitoring. Poultry target positioning: through the visible light image (RGB image), the texture analysis and feature point extraction algorithm is used to identify the poultry target in the high occlusion area of feathers, and the contour edge and key joint position are extracted. Motion trajectory analysis: for the claw part (i.e. the toe joint area) of the poultry, the swing trajectory is calculated through the time series image analysis, and the breakpoint (i.e. the continuous displacement vector is lower than the threshold or the mutation position) is identified as a potential indicator of behavior abnormalities. Displacement and temperature distribution extraction: combine the infrared thermal imaging image to analyze the annular temperature gradient direction of the anus area. This temperature distribution is used to infer the heat conduction path in the body, which is the basis for subsequent behavior inversion reasoning. Gradient direction refers to the maximum direction of temperature value change with spatial position, which is used to describe the heat diffusion trend. Behavior inversion module, this module is used to infer the hidden behavior state of the target limb when it is occluded by feathers or the image is incomplete, and to reconstruct the continuous motion sequence. Heat conduction delay reasoning: based on the temperature gradient direction around the anus in the infrared image, the physical characteristics of the time delay of heat conduction in muscle tissue are used, combined with the heat conduction path model, to infer the joint flexion angle of the occluded leg (i.e. the change amplitude of joint range of motion). Spatial consistency constraint: use the motion direction vector of the surrounding poultry individuals in the same time window to establish a local group motion direction constraint condition. If the target poultry deviates significantly from the adjacent individuals, it is considered as an abnormal motion trajectory. Gait sequence repair: integrate the inferred angle and trajectory breakpoint to generate a complete gait data sequence through interpolation or fitting algorithm (such as cubic spline), realize the continuous reconstruction of the motion behavior in the occluded area. Fusion analysis module, this module is used to integrate the information from different image sources (visible light + infrared) and different behavior dimensions (motion, posture, temperature), unify the time reference, and extract individual behavior features. The specific functions are as follows: time axis alignment and sequence matching: take the timestamp of the foot landing point in the gait sequence as the reference axis, and resample and align (time series synchronization) the body tilt angle sequence (obtained from the spine posture change) in the visible light image. Temperature difference fusion: identify the left and right foot bottom areas in the infrared image, extract the thermal radiation contour and calculate the average temperature difference of the foot center, generate a temperature difference sequence, which is used to reflect the foot load imbalance of poultry during the gait support phase. Spatial trajectory analysis: based on the depth map (obtained by structured light or TOF depth camera), the three-dimensional position distribution model of the chicken group is reconstructed, and then the dynamic deviation coefficient of individual relative to the group centroid in the motion cycle is calculated, which reflects the out-of-group degree of individual behavior and is a key parameter for identifying abnormal individuals. Behavior decision module, this module is responsible for converting the foregoing fusion features into behavior classification results, judging whether the behavior is abnormal and the type of abnormality, and evaluating the credibility of the judgment result.The implementation modes thereof include: environment parameter modeling: according to the collected ground temperature and ammonia concentration information, a two-dimensional environment factor space is constructed, and a stress behavior threshold surface model (for example, fitting is performed by using a radial basis function kernel) is established by using historical samples, and the model is used to divide the discrimination boundary of normal behavior and stress behavior. Feature coupling analysis: three core behavior features are extracted and combined: gait support phase time length (representing single foot support duration), body inclination angle (reflecting body balance state) and motion trajectory deviation coefficient (individual group deviation degree), forming a three-dimensional feature vector. Classification and confidence output: a pre-trained classifier is used to classify the three-dimensional feature vector, outputting an abnormal behavior type (such as mechanical movement disorder or environmental stress adaptation behavior), and calculating the confidence according to the distance of the feature point to the classification boundary.

[0026] Specifically, the processing process of the occlusion target perception module includes the following contents: frame-by-frame scanning of the visible light video stream, identification of the claw joint region in the feather gap by a multi-scale texture matching algorithm, analysis of the gray scale distribution of the visible light image, and output of the feather covering density level; by tracking the position change of each joint, the claw swing trajectory is constructed, and the displacement vector between adjacent frames is calculated, and when the displacement vector exceeds the preset continuity threshold, it is marked as an interruption point; the three-dimensional space coordinates of the head exposure area are calculated by fusing the binocular camera view angle, the pitch angle and the yaw angle of the head are calculated according to the three-dimensional coordinate changes of the continuous frames, and the displacement vector is calculated, and the displacement direction and amplitude are quantified by combining the angle change rate.

[0027] In the embodiment, the processing process of the occlusion target perception module is as follows: the multi-scale texture matching algorithm identifies the claw joint region by frame-by-frame scanning of the visible light video stream. For the identification of the claw joint in the feather gap, a multi-scale texture matching algorithm is used. Based on the local texture features of the image, a scale space is constructed, and the texture pattern of the claw joint is detected layer by layer from coarse to fine, improving the detection accuracy in the occlusion environment. This step mainly realizes the positioning of the claw detail area, and outputs the two-dimensional image coordinates of the claw joint. The feather covering density level is evaluated by analyzing the gray scale distribution function I(x, y) of the visible light image, calculating the entropy H of the gray scale histogram in the region to represent the complexity and density of the feather covering: ; parameter description: G: gray scale; P(g): proportion of pixels with gray value g; H: gray entropy, the larger the value, the more complex the occlusion texture and the higher the feather density. According to the threshold range of H, the feather covering density level (low, medium and high) is divided. The claw swing trajectory construction and interruption point marking use a tracking algorithm to extract the two-dimensional coordinate sequence of each claw joint node , and calculate the displacement vector between adjacent frames: ; wherein, : the position of the th claw joint in the th frame; Adjacent frame displacement vector. Define continuity threshold When , it is considered as a trajectory breakpoint, indicating that the claw movement information is abnormal or occlusion causes tracking failure. Head exposure area three-dimensional space coordinate calculation uses the principle of binocular camera stereo vision, based on the matching of corresponding points in two views, to recover the three-dimensional space coordinates of the head exposure area . Its calculation formula depends on the camera internal and external parameters and the disparity mapping, and the specific formula is as follows: ; Parameter description: : Camera focal length : Binocular baseline length : Left and right camera disparity : Left eye pixel coordinates : Camera principal point coordinates : Head three-dimensional coordinates. Head pitch angle and yaw angle calculation is based on the sequence of head three-dimensional coordinates of consecutive frames, to calculate the pitch angle (Pitch) and yaw angle (Yaw) of the head, which are defined as the rotation angles around the horizontal and vertical axes respectively. Calculate the angular rate of change of head movement in unit time, the formula is as follows: ; Wherein, ; Displacement vector and displacement direction and amplitude quantification Head displacement vector is defined as: ; Its amplitude is calculated as: ; Combined with the angular rate of change (approximately calculated by difference), the displacement direction is vectorized and quantitatively analyzed to determine the spatial direction and dynamic amplitude of head movement.

[0028] Specifically, the processing process of the occlusion target perception module further includes the following contents: dynamically adjusting the segmentation threshold according to the mapping relationship between the predefined environment temperature and the infrared image segmentation threshold, binarizing and segmenting the infrared image through the adjusted threshold, filling the holes combined with morphological closing operation, obtaining the connected region of the anus region; calculate the temperature gradient of the contour edge of the anus region, extract the edge point with the maximum gradient value and connect it into a line, calculate the direction angle of the line; according to the pre-established mapping relationship between the gradient direction angle and the leg muscle contraction, convert the current gradient direction angle into the corresponding leg muscle contraction state, and output the muscle activation intensity code.

[0029] In this embodiment, the dynamic adjustment of the infrared image segmentation threshold is to adapt to the influence of the change of the environment temperature on the response range of the infrared image. The system introduces an environment temperature segmentation threshold mapping function φ(E), and dynamically calculates the segmentation threshold Tθ through the function: ; Parameter description: T0: the adaptive segmentation threshold of current infrared image; E: the measured value of current ambient temperature; a1, b1, g1, d1: calibration coefficients of fitting mapping relationship, which are obtained by training based on historical sampling data. This formula is used to construct the nonlinear response between the segmentation threshold and the ambient temperature, in order to overcome the problem of poor boundary recognition stability of the fixed threshold in practical application. The infrared image is segmented by thresholding and the morphological closing operation to obtain the preliminary target region mask: ; Subsequently, the morphological closing operation is performed on the binary image B(u, v): ; Parameter description: B(u, v): original binary image; K: structure element, usually an elliptical kernel; o: dilation operation; o: erosion operation; B'(u, v): connected region image after filling holes, used for anus region candidate extraction. The anus region contour edge temperature gradient calculation is based on the connected region after the closing operation, extracts its boundary contour R, and calculates the temperature gradient G_n in the normal direction on the boundary, which is defined as follows: ; Parameter description: q e R: pixel point on the contour; Gradient vector of the infrared image at point q; : unit normal direction at point q; : temperature gradient in the normal direction, indicating the strength of heat diffusion or aggregation. Extract the point set that satisfies: from all contour points, and record it as the high gradient edge point sequence. Gradient edge point connection and direction angle calculation connects the maximum gradient point set in spatial order to form the edge main axis segment, and sets the two end points as ( , ), ( , ).The direction angle ψ can be expressed as: ; Parameter description: , ), , : coordinates of the two end points of the connection; ψ: gradient main axis direction angle, indicating the offset direction of the temperature ring distribution; ε: a small amount of compensation term to prevent the denominator from being zero. The angle value is used for further analysis of the corresponding physiological state. Gradient direction angle mapping to muscle activation state According to the pre-established gradient direction angle muscle contraction response mapping function Θ(ψ), the direction angle is converted into muscle activation intensity coding , which is in the following form: ; Parameter description: ψ: temperature gradient main axis direction angle; : leg muscle activation intensity encoding, as the quantitative output of neuromuscular response; η1,η2: amplitude factors, adjusting the amplitude of activation response; ω1,ω2: frequency factors, corresponding to the angular sensitivity of muscle group response; φ1,φ2: phase offset, used to calibrate the activation interval. The final will be one of the important input parameters of the "joint flexion angle inference" in the behavior inversion module.

[0030] Specifically, the behavior inversion module includes the following: determine the heat conduction path of the covered leg according to the muscle activation intensity encoding, and measure the delay time of heat conduction; load the corresponding heat conduction rate correction coefficient according to the feather covering density level, and calculate the actual heat conduction delay time by applying the conduction rate correction coefficient; calculate the joint flexion angle according to the pre-marked delay time and joint flexion angle relationship curve.

[0031] In this embodiment, the length of the heat conduction path is determined as the heat flow direction from the center area of the anus temperature to the muscle group area of the covered leg, and the path length can be estimated by three-dimensional coordinate point cloud: ; Parameter description: : the length of the heat conduction path from the center of the anus to the heat response center of the covered leg; : the three-dimensional coordinates of the first space point on the path; : the total number of discrete points of the heat path; : the Euclidean distance calculation symbol. The path is estimated by the main direction of the isotherm in the infrared image space heat distribution. The actual delay time of heat conduction is measured by recording the frame number required for the temperature of the muscle group area to reach the inflection point (such as the maximum first derivative) in the infrared image, denoted as , which reflects the time response of temperature from the source to the end point. II. Feather covering correction factor loading and delay correction Since the feather covering affects the heat spreading rate on the body surface, the correction coefficient should be loaded according to the covering density level , and the actual heat conduction delay time : ; Parameter description: : corrected heat conduction delay time; : original measured heat conduction delay time; : feather covering density level, which is a continuous variable with a value of 0 to 1; : feather thermal resistance correction coefficient, reflecting the amplification ratio of feather to heat conduction delay; : empirical index, describing the nonlinear strength of feather covering and heat conduction delay, which is generally calibrated by experiment. According to the pre-marked "heat delay-flexion angle" relationship function , the corrected delay time Mapping of knee or ankle flexion angle to current frame ; Parameter Description: Inferred leg joint flexion angle in angle value; : Thermal conduction delay time after feather shading correction; : Calibration fitting coefficient, obtained by least squares fitting after collecting multiple samples through experiments. This mapping curve reflects the indirect reaction characteristics of muscle group temperature response to movement posture.

[0032] Specifically, the behavior inversion module further includes the following: identifying the start and end frame positions of the claw trajectory breakpoint, obtaining the moving direction vector of the adjacent poultry within a radius set range centered on the target poultry, and fusing the joint flexion angle and the neighbor moving direction vector to generate a spatial constraint direction; according to the claw trajectory trend before the breakpoint and the spatial consistency constraint direction, using cubic spline interpolation to generate a repaired trajectory in the interruption interval, connecting the repaired trajectory with the original trajectory to form a complete gait sequence.

[0033] In the embodiment, the identification of the frame segment in the interruption interval is performed by detecting the position continuity of the claw of the target poultry between frames, and defining the spatial trajectory thereof as: ; wherein the consecutive frames satisfy the following distance threshold judgment: ; Parameter Description: : Original claw trajectory set of the target poultry; : The frame claw two-dimensional image coordinates; : Consecutive frame trajectory jump distance threshold, used to identify the trajectory breakpoint; : Total number of frames; : Frame index. After detecting multiple consecutive jumps, the start frame and the end frame of the interruption interval can be defined, i.e., the interruption interval is . The neighbor direction vector extraction takes the position of the target poultry in the frame before the interruption interval as the center point , sets the radius , and retrieves the center position trajectory of the adjacent individuals in the image coordinate system: ; calculate the moving direction vector of each neighbor in the interruption interval: ; aggregate to form the average neighbor direction vector: ; Parameter Description: : Poultry individual set in the neighborhood; : Center coordinates of the neighbor individual in frame ; Average moving direction vector in the neighborhood: ​​​: Number of neighbors. The current frame flexion angle direction unit vector is fused with neighbor direction vectors to generate the repaired constraint direction: ; Parameter description: : Spatial consistency constraint direction vector; : Fusion coefficient, reflecting the weight balance of the current individual dominance and group consistency; : Local claw movement direction unit vector determined by joint angle. The three-spline interpolation is used to repair the interrupted trajectory, and the endpoints before and after the interruption are set as and , and the number of frames in the interruption interval is . The intermediate points are inserted in a three-spline manner: ; and the endpoint derivative is adjusted through direction constraint, so that: ; and finally: ; Parameter description: : Three-spline expression of the repaired curve; : Spline fitting coefficient; : Final fused and repaired complete gait trajectory; : Total number of frames.

[0034] Specifically, the fusion analysis module includes the following: extracting the foot landing time stamp in the repaired gait sequence as the reference time axis, resampling the spine curvature angle change sequence in the visible light image to the time axis, eliminating the collection time error of the gait sequence and the body tilt angle sequence through the dynamic time warping algorithm; segmenting the thermal radiation profile of the left and right foot bottom region in the infrared image, calculating the average temperature difference of the foot center region to generate a temperature difference sequence, and aligning the temperature difference sequence with the gait support phase starting time point through phase delay compensation.

[0035] In the embodiment, the reference time axis is established and the angle sequence resampling step extracts the foot landing time stamp as the time anchor point from the repaired gait sequence to extract the landing point coordinate sequence , and determines the time of the foot bottom contacting the ground event as: ; Parameter description: : Repaired gait trajectory sequence; : The th landing point position; : The time stamp set of the corresponding landing point event; : The frame time of the th landing. The spine angle sequence is resampled to the gait time axis in the visible light image sequence to obtain the spine angle (such as the thoracolumbar angle) sequence , to ensure alignment with the gait data, it is resampled to the time axis Non-uniform resampling on the upper body: ; where the resampling function is implemented based on interpolation, and then the two sets of time series data are aligned using dynamic time warping algorithm to obtain the paired path: ; this path is used to correct the misalignment of gait and posture sequences in acquisition frequency and phase. Infrared image foot region segmentation and temperature extraction, through temperature gradient enhancement and region growing algorithm, the left and right foot infrared image contours are segmented, and the foot center area is marked as and , respectively, to calculate the average temperature: ; generate thermal difference sequence: ; parameter description: : left and right foot center area pixel set; : the radiation temperature value of the pixel point in the infrared image; : the foot temperature difference at the th landing. The temperature difference sequence is aligned with the support phase time. Considering that infrared imaging has thermal inertia, the temperature response has a delay relative to the gait event, and assuming that the delay is , the temperature difference sequence is phase delayed and compensated: ; where the delay can be obtained by minimizing the time series deviation function of the temperature difference and the start event of the support phase: ; parameter description: : support phase flag function (e.g. support phase is 1, swing phase is 0); : compensated temperature difference value.

[0036] Specifically, the fusion analysis module further includes the following contents: reconstructing the three-dimensional spatial distribution model of the chicken group through the point cloud data of the depth camera, estimating the body size index of each poultry based on the visible light image; calculate the weighted group centroid coordinates with the square of the body size index as the weight, sample the individual position coordinates at equal intervals within the gait cycle, and calculate the Euclidean distance from each sampling point to the weighted centroid trajectory in real time. The ratio of the standard deviation to the mean of the Euclidean distance sequence is taken as the dynamic deviation coefficient.

[0037] In this embodiment, the three-dimensional group model construction and weighted centroid calculation, the three-dimensional spatial model reconstruction uses the depth camera to obtain each frame of point cloud data , where the point set of each poultry body surface is , through clustering and posture filtering, the first poultry is identified and segmented. The body size index is estimated according to the outline frame area of each poultry in the visible light image, and the length of the long axis of the torso and the transverse width are extracted, and the body size index is defined as: ; parameter description: : No. Body mass index of poultry; Major axis length (length in the main direction); Horizontal width; The weighting exponential factor (empirical value, generally taken as ) controls the nonlinear convergence of body shape. Between). Weighted centroid trajectory calculation: Let the first centroid in the gait period be... The coordinates of all poultry in the frame in three-dimensional space are Then the weighted group centroid coordinates of this frame are: Parameter description: : No. The weighted centroid coordinates of the frame; : No. The three-dimensional coordinates of a poultry; : Number of poultry individuals in the current frame; Body size index. This weighting method enhances the contribution of larger individuals to the centroid shift, improving accuracy. The dynamic deviation coefficient of the individual trajectory is calculated by sampling the individual's motion trajectory within a complete gait cycle, and then using the target individual's position sequence {x1(t), x2(t), ..., x...}. n (t)} and the corresponding weighted centroid sequence {x1(c),x2(c),…,x n (c) Alignment. Euclidean distance sequence calculation: Calculate the Euclidean distance between the target individual and the weighted centroid in each sampling frame: Parameter description: :No. Frame offset distance; : Three-dimensional coordinates The component values ​​correspond to the target individual and the centroid, respectively. : Corresponds to X, Y, Z in three-dimensional space. Dynamic deviation coefficient calculation is defined as the dynamic deviation coefficient within the gait cycle. The ratio of the standard deviation to the mean of the Euclidean distance sequence: Parameter description: Dynamic deviation coefficient; Average offset distance across all frames; Number of sampling frames. This coefficient can effectively measure whether individual behavior tends to conform to the consistency of group movement, thus reflecting potential behavioral abnormalities.

[0038] Specifically, the root behavior decision module includes the following: establishing a two-dimensional parameter coordinate system for ground temperature and ammonia concentration; deploying data acquisition grid points in the parameter space; recording the support phase duration and body tilt angle range of normal poultry at each grid point; applying radial basis function kernels to fit the support phase duration threshold surface; establishing a body tilt angle tolerance boundary model through polynomial regression; acquiring the current environmental parameter coordinate points in real time; and interpolating and outputting the stress behavior threshold in the threshold surface and boundary model.

[0039] In this embodiment, two-dimensional environmental parameter modeling and threshold space construction, ground temperature and ammonia concentration two-dimensional parameter space is constructed to ground temperature and ammonia concentration As an environmental factor, it constitutes a two-dimensional environmental coordinate system: ; Parameter description: : current frame environmental parameter point; : ground temperature; : ammonia concentration. In this coordinate system, the experimental data sampling grid , record the characteristics of each grid point on the normal poultry behavior: support phase time length mean and standard deviation , ; upper and lower limit of body inclination angle . Establish support phase time length threshold surface (RBF fitting) using radial basis function to fit the trend of support phase time length in parameter space, get the threshold surface with strong continuity. The fitting model is expressed as follows: ; Parameter description: : support phase time length threshold under the current environment; : weight coefficient of the th kernel function; : kernel function (commonly used Gaussian kernel ); : the environmental coordinates of the th sampling grid point; : the total number of grid points. Optimize parameters by minimizing the sum of squares of residuals. Posture tolerance boundary fitting and stress determination, establish body inclination angle tolerance boundary model (polynomial fitting) to take the body inclination angle tolerance range of each grid point as sample, establish the upper and lower limit boundary model of inclination angle. Use quadratic polynomial regression fitting for minimum boundary and maximum boundary respectively: ; ; Parameter description: : the lower and upper limits of the tolerance angle under the current environmental parameters; : polynomial model coefficients, fitted. Real-time stress behavior threshold extraction and determination system in real-time perception of the current frame environment state , and through the above model interpolation to get the current threshold: support phase time length threshold: inclination angle tolerance interval: Compare the current individual support phase time length and body inclination angle with the above interval: if and This was determined to be abnormal behavior in response to environmental stress. This module constructs a multimodal perception-driven real-time behavior state interpretation system through an environmental behavior coupling response mechanism.

[0040] Specifically, the root behavior decision module also includes the following: normalizing the support phase duration based on the environmental stress behavior threshold curve, standardizing the tolerance boundary of the body tilt angle, and implementing density adaptive scaling of the dynamic deviation coefficient of the movement trajectory; constructing a three-dimensional feature vector containing the normalized support ratio, standardized tilt angle, and scaled deviation coefficient; mapping the feature vector to the decision space through a pre-trained classifier, calculating the confidence level based on the distance from the feature vector to the classification boundary, determining mechanical movement disorder when the support ratio continuously exceeds the threshold and the deviation coefficient is abnormal, and marking environmental adaptive behavior when the tilt angle momentarily exceeds the standard but other features are normal.

[0041] In this implementation scheme, feature normalization and scaling processing, and support phase duration normalization processing are based on the stress behavior support phase duration threshold calculated in stage 1. Regarding the current observation duration Perform proportional normalization: Parameter description: : Normalized support ratio; : Support phase duration of the current frame; Support phase threshold under current conditions. Body tilt angle tolerance standardization based on predicted angle tolerance range. , the current angle Perform boundary standardization: Parameter description: Standardized tilt angle index (0~1 if it falls within the range); The currently observed body tilt angle. Motion trajectory deviation coefficient density adaptive scaling assumes the dynamic deviation coefficient of the current frame is... Establish a kernel density estimation function for the deviation coefficient distribution of all training samples: The density adaptive scaling factor is further defined as follows: Parameter description: : Deviation coefficient scaling amount; : Kernel density function of deviation coefficient; : No. The deviation coefficient of each sample; Kernel functions (such as Gaussian); Bandwidth factor; : A regular constant to prevent the denominator from being zero; Total number of training samples. Three-dimensional feature space mapping and classifier inference, constructing a behavioral feature triplet vector: combining the above three indicators to form the behavioral feature vector of the current frame. ; Parameter Description: : 3D feature vector of current frame, representing support strength, posture stability, and motion coordination. Based on the classifier mapping to the decision space, the feature vector is mapped to the behavior category label through the pre-trained binary or multi-class classifier model (such as support vector machine SVM, random forest RF, etc.). ; and the distance of the feature point to the classification boundary (support vector machine) is calculated: ; Parameter Description: : trained classifier model; : normalized distance of the feature point to the classification boundary; : normal vector of the classification hyperplane; : offset. Convert to a confidence index : ; wherein is a positive hyperparameter for controlling the confidence growth rate. Rule-based reasoning determines the behavior type, and the behavior type is inferred based on the threshold combination rule: if and , it is judged as a mechanical movement disorder behavior; if , but the other two indicators are in the normal interval, it is marked as an environmental adaptability behavior; the rest is considered as a normal or transitional behavior state. The final system outputs the behavior label and the confidence for subsequent alarm, intervention or record.

[0042] In summary, the present application has at least the following effects:

[0043] ​​The poultry behavior abnormality real-time monitoring system based on multi-modal image fusion fuses visible light and infrared image information, adopts multi-scale texture analysis and temperature gradient direction extraction method, realizes accurate identification and state perception of key physiological regions under high feather shielding, and effectively enhances target detection capability in natural shielding environment. Based on the temperature conduction delay around the anus and the corresponding muscle contraction state, combined with the claw trajectory interruption and the motion direction of adjacent individuals, a spatial consistency constraint condition is constructed, and the missing gait sequence is completed by using interpolation technology, so that the actual motion behavior of the shielding leg is accurately reversed, and the explanation ability of the system to non-explicit behavior is improved. A unified time axis is constructed to synchronize the trunk tilt sequence in the visible light image and the thermal infrared foot temperature difference sequence, and a group spatial distribution model based on three-dimensional point cloud reconstruction is introduced, which dynamically quantifies the deviation degree of individuals to the centroid trajectory, and provides complete space-time and group behavior context information for behavior pattern recognition. By introducing the two-dimensional environmental parameters of ground temperature and ammonia concentration, a behavior threshold surface based on radial basis function and regression model is constructed, and three-dimensional behavior features such as normalized support phase length, standardized trunk posture and density scaling deviation coefficient are fused, so that mechanical movement disorders and adaptive behaviors can be distinguished in real time under dynamic environmental conditions, and the intelligent discrimination accuracy and confidence output ability of the system are improved.

[0044] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware aspects. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) having computer usable program code embodied thereon.

[0045] The application is described with reference to flowcharts and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The means for implementing the functions specified in a flow or multiple flows and / or blocks

[0046] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0047] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0048] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to encompass within their scope all possible variations and modifications of the preferred embodiments. 1

[0049] It is apparent that a person skilled in the art can make various changes and modifications to the application without departing from the spirit and scope thereof. Thus, if these modifications and changes fall within the scope of the claims and their equivalents, it is intended to include them in the application.

Claims

1. A real-time monitoring system for abnormal poultry behavior based on multimodal image fusion, characterized in that, It includes the following modules: occlusion target perception module, behavior inversion module, fusion analysis module, and behavior decision-making module; The occlusion target perception module is used to locate poultry targets in areas with high feather occlusion through visible light images, extract the interruption point of their claw swing trajectory and the displacement vector of the exposed area of ​​their head, and simultaneously analyze the direction of the anal temperature ring distribution gradient based on infrared images. The behavior inversion module is used to infer the joint flexion and extension angle of the covered leg based on the thermal conduction delay characteristics of the anal temperature gradient direction, and generate the repaired gait sequence by combining the interruption point of the claw trajectory and the spatial consistency constraint of the movement direction of adjacent poultry groups. The fusion analysis module matches the repaired gait sequence with the changes in body tilt angle in the visible light image on the time axis, and simultaneously fuses the temperature difference features of the left and right feet in the infrared image. Based on the three-dimensional position distribution of the flock, it calculates the dynamic deviation coefficient of the individual movement trajectory from the center of mass of the flock. The behavior decision module establishes an environmental stress behavior threshold curve based on ground temperature and ammonia concentration. By coupling the support phase duration, body tilt angle, and dynamic deviation value of the movement trajectory in the gait sequence, it distinguishes between mechanical movement disorders and environmental adaptive behaviors, and outputs the abnormal behavior type and confidence level.

2. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 1, characterized in that: The processing steps of the occlusion target perception module include the following: The visible light video stream is scanned frame by frame, and the claw joint region in the gaps between feathers is identified by a multi-scale texture matching algorithm. The grayscale distribution of the visible light image is analyzed, and the feather occlusion density level is output. By tracking the positional changes of each joint, the claw swing trajectory is constructed, and the displacement vector between adjacent frames is calculated. When the displacement vector exceeds the preset continuity threshold, it is marked as an interruption point. The system calculates the three-dimensional spatial coordinates of the exposed area of ​​the head by fusing the perspective of the binocular camera, calculates the pitch and yaw angles of the head based on the changes in the three-dimensional coordinates of the continuous frames, calculates the displacement vector, and quantifies the displacement direction and magnitude by combining the rate of change of angle.

3. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 2, characterized in that: The processing of the occlusion target perception module also includes the following: The segmentation threshold is dynamically adjusted according to the predefined mapping relationship between ambient temperature and infrared image segmentation threshold. The infrared image is then binarized and segmented using the adjusted threshold. Morphological closing operations are then used to fill the holes, resulting in the connected regions of the anal region. Calculate the temperature gradient along the edge of the anal region contour, extract the edge points with the largest gradient values ​​and connect them into a line, and calculate the direction angle of the line. Based on the pre-established mapping relationship between gradient direction angle and leg muscle contraction, the current gradient direction angle is converted into the corresponding leg muscle contraction state, and the muscle activation intensity code is output.

4. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 3, characterized in that: The behavior inversion module includes the following: The heat conduction path of the covered leg was determined based on the muscle activation intensity coding, and the heat conduction delay time was measured. Based on the feather coverage density level, apply the corresponding heat conduction rate correction factor, and then calculate the actual heat conduction delay time using the conduction rate correction factor. The joint flexion and extension angles are calculated based on the pre-calibrated curve showing the relationship between the delay time and the joint flexion and extension angles.

5. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 4, characterized in that: The behavior inversion module also includes the following: Identify the start and end frame positions of the claw trajectory interruption point, and obtain the movement direction vectors of neighboring poultry within a set radius with the target poultry as the center. Combine the joint flexion and extension angles with the neighbor movement direction vectors to generate spatial constraint directions. Based on the claw trajectory trend and spatial consistency constraint direction before the interruption point, cubic spline interpolation is used to generate a repair trajectory for the interruption interval. The repair trajectory is then connected to the original trajectory to form a complete gait sequence.

6. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 5, characterized in that: The fusion analysis module includes the following: The foot landing time stamps in the repaired gait sequence are extracted as the reference time axis. The spinal curvature angle change sequence in the visible light image is resampled to this time axis. The acquisition time error between the gait sequence and the body tilt angle sequence is eliminated by dynamic time warping algorithm. The thermal radiation profiles of the left and right foot regions in the infrared image are segmented, the average temperature difference of the center of the foot is calculated to generate a temperature difference sequence, and the temperature difference sequence is aligned with the starting time point of the gait support phase by phase delay compensation.

7. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 6, characterized in that: The fusion analysis module also includes the following: A three-dimensional spatial distribution model of the chicken flock was reconstructed using point cloud data from a depth camera, and the body size index of each poultry was estimated based on visible light images. The weighted centroid coordinates of the population are calculated using the square of the body mass index as the weight. Individual position coordinates are sampled at equal intervals within the gait cycle. The Euclidean distance from each sampling point to the weighted centroid trajectory is calculated in real time. The ratio of the standard deviation to the mean of the Euclidean distance sequence is used as the dynamic deviation coefficient.

8. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 7, characterized in that: The root behavior decision module includes the following: Establish a two-dimensional parameter coordinate system for ground temperature and ammonia concentration, and set up data acquisition grid points in the parameter space. Record the support phase duration and body tilt angle range of normal poultry at each grid point. A radial basis function kernel is used to fit the support phase duration threshold surface. A body tilt angle tolerance boundary model is established through polynomial regression. The coordinate points of the current environmental parameters are obtained in real time. The stress behavior threshold is interpolated and output in the threshold surface and boundary model.

9. The real-time monitoring system for abnormal poultry behavior based on multimodal image fusion according to claim 8, characterized in that: The root behavior decision module also includes the following: The duration of the support phase is normalized based on the environmental stress behavior threshold curve, the tolerance boundary of the body tilt angle is standardized, and the density adaptive scaling is applied to the dynamic deviation coefficient of the motion trajectory. Construct a three-dimensional feature vector containing normalized support ratio, standardized tilt angle, and scaling deviation coefficient; The pre-trained classifier maps feature vectors to the decision space, and calculates confidence based on the distance from the feature vector to the classification boundary. When the support ratio continuously exceeds the threshold and the deviation coefficient is abnormal, mechanical movement disorder is identified. When the tilt angle instantaneously exceeds the standard but other features are normal, environmental adaptive behavior is marked.

Images