A Non-Contact Method and System for Detecting Respiratory Rate in Reclining Cattle Based on RGB-D

Abstract

This invention relates to a non-contact method and system for detecting the respiratory rate of recumbent cattle based on RGB-D technology, belonging to the field of computer vision-based intelligent cattle farming technology. Addressing the problems of low detection accuracy and stringent environmental requirements in current methods for detecting cattle respiratory rate, the method of this invention includes: non-contactly acquiring RGB-D video stream data of a recumbent cattle to obtain an RGB visible light video stream and a depth stream; locating key parts of the recumbent cattle; defining a key target area on the abdomen based on the key parts; registering the RGB visible light video stream and the depth video stream according to their positional coordinate correspondence; obtaining the average depth value of the key target area on the abdomen; plotting a respiratory change curve based on the changes in the average depth value; and detecting and statistically analyzing the peak values ​​of the respiratory change curve to obtain the number of breaths. This invention's detection method is effective and reliable, providing a new approach for realizing non-contact intelligent cattle respiratory rate detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision-based intelligent cattle farming technology, and in particular to a non-contact method and system for detecting the respiratory rate of lying cattle based on RGB-D. Background Technology

[0002] Traditional methods for detecting bovine respiratory rate mainly rely on manual detection. This method not only requires highly specialized personnel but is also inefficient and cannot meet the needs of large-scale cattle farms for detecting respiratory rate. With the development of sensor technology, respiratory signals from cattle are acquired based on the characteristics of different sensors, mainly divided into wearable (i.e., contact) respiratory rate detection and non-contact respiratory rate detection.

[0003] Wearable detection devices measure the respiratory rate of cattle in a contact manner, providing continuous and accurate measurements. However, this method relies heavily on manual inspection, requiring highly skilled personnel and resulting in low efficiency. This makes it unsuitable for the respiratory rate monitoring needs of large-scale farms. Furthermore, the installation and maintenance of the hardware are complex, and prolonged wear can cause stress in cattle, reducing their comfort and potentially leading to destructive behavior that affects the accuracy of the respiratory rate measurement. Non-contact detection methods, primarily based on thermal infrared imaging for nasal temperature detection, RGB visible light image processing for abdominal movement detection, and lidar for abdominal distance detection, combine with deep learning technology to achieve intelligent and automated respiratory rate detection. These methods minimize interference with normal animal behavior and provide good respiratory rate detection results. Existing research has demonstrated the effectiveness of non-contact detection methods; however, in practical applications, factors such as cattle limb movements and ambient lighting conditions still present limitations that require improvement.

[0004] For example, in non-contact RGB optical flow methods, cattle are easily disturbed by mosquitoes in summer, causing them to shake their abdomens to expel the insects. This can lead to abnormal changes in the abdominal optical flow field in the image, thus affecting the accuracy of respiratory rate detection. Moreover, this optical flow method requires the image brightness in the environment to remain constant during video acquisition. In detection methods based on lidar technology, it can currently only be used in milking scenarios. Furthermore, the milking process can also cause slight stress reactions in cattle, and the detected respiratory rate may differ from the value in the calm state, affecting the accurate judgment of the cattle's health status.

[0005] Therefore, this study proposes a non-contact method for detecting the respiratory rate of cattle lying on their sides based on RGB-D. This method detects the respiratory rate of cattle through non-contact technology. It can detect abnormal respiratory rates in cattle when they experience heat stress due to abnormal temperature and humidity changes in the breeding environment or when individual cattle develop respiratory diseases. Farmers can take appropriate artificial interventions by detecting abnormal changes in the respiratory rate of cattle, thereby reducing economic losses in cattle farms. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, this invention proposes a non-contact method and system for detecting the respiratory rate of cattle in a side-lying position based on RGB-D. By integrating RGB-D depth data with computer vision technology, a non-contact method for detecting the respiratory rate of cattle in a side-lying position is constructed to solve the problems existing in the background technology.

[0007] To achieve the above technical objectives, the present invention adopts the following technical solution: In a first aspect, this invention discloses a non-contact method for detecting the respiratory rate of a laterally recumbent cow based on RGB-D, comprising the following steps: S1: Collect RGB-D video stream data of lying cattle in a non-contact manner, and preprocess the RGB visible light video stream and depth video stream in the RGB-D video stream data respectively; S2: Based on the preprocessed RGB visible light video stream, locate the key parts of the side-lying cattle, and combine the key parts to define the key target area of ​​the abdomen; S3: Based on the positional coordinate correspondence between the RGB visible light video stream and the depth video stream, registration is performed to obtain the average depth value of the key target area in the abdomen; S4: Plot the respiratory change curve based on the average depth change of the key target area in the abdomen, and use the AMPD algorithm to detect peaks in the respiratory change curve, count the number of peaks, and obtain the number of breaths.

[0008] As a further preferred option, step S2, which involves locating key areas of the lying-on-side cattle and defining the key target area of ​​the abdomen based on these key areas, includes the following specific steps: S21: Using the YOLOv8n-Pose network structure, four key points were located on the neck and forelimbs of the side-lying cattle. S22: Based on the positions of four key points, dynamically select the coordinate information of the abdomen of the lying cow to obtain the key target area of ​​the abdomen.

[0009] A further preferred embodiment is that, in step S22, the step of obtaining the key target region of the abdomen includes: Connect the two opposite corners of the four key points; Determine the intersection of the two lines, obtain the rectangular area centered on the intersection, and use the rectangular area as the key target area of ​​the abdomen.

[0010] Preferably, before step S3, if it is determined that the current key abdominal target area is occluded, a sliding window is set for the key abdominal target area enclosed by the four currently selected key points, and a new key abdominal target area is obtained by adjusting the rectangular area.

[0011] As a further preferred embodiment, step S3, which involves obtaining the average depth value of the key target region in the abdomen, includes: S31: Acquire an RGB visible light image and a depth image containing depth and distance information; S32: Based on the coordinate system transformation between the RGB visible light image and the depth image, registration is performed to obtain the depth distance value between the key target area on the abdomen in the RGB visible light image and the same coordinate position in the depth image.

[0012] A further preferred embodiment is that, in step S32, the registration step based on the coordinate system transformation between the RGB visible light image and the depth image includes: The expression for restoring the pixels of a depth image to the coordinate system of the depth image, and restoring the coordinates of the points in the coordinate system of the depth image to the world coordinate system, is as follows: In the formula, Represents the coordinates of a point in the coordinate system of a depth image. Z represents the pixel in the depth image, and Z represents the depth value of the pixel. This represents the intrinsic parameter matrix of the depth camera; The coordinates of points in the world coordinate system are transformed to the coordinate system of the color image using a Euclidean transformation matrix. The expression is as follows: In the formula, The depth point is represented in the coordinate system of the color image, and T is the Euclidean transformation matrix. This is the transformation matrix for converting from the world coordinate system to the color camera coordinate system. R is the transformation matrix from the coordinate system of the depth image to the world coordinate system, R is the rotation matrix, is unit orthogonal, and t is the translation amount of the axis. The depth points in the coordinate system of the color image are mapped to the color image plane with Z=1, and the Z-axis is normalized to complete the task of aligning the depth image to the RGB visible light image. The expression is as follows: In the formula, This indicates the pixel coordinates of the depth point on the RGB visible light image. This indicates the intrinsic parameters of the RGB camera.

[0013] Furthermore, the steps for obtaining the average depth value of the key target region on the abdomen include: Obtain the ratio α of the pixel values ​​of the depth image to the physical distance; Traverse all pixels in the key target area of ​​the abdomen and calculate the distance value represented by each pixel according to the ratio α; The average distance value of the key target region in the abdomen is calculated based on the distance value represented by each pixel.

[0014] As a further preferred option, step S4, which involves using the AMPD algorithm to detect peak values ​​in the respiration change curve, includes: S41: The sliding detection window is determined based on the AMPD algorithm. The calculation formula for confirming the window size is as follows: In the formula, S(k) represents the local maximum weight within each window, and k is the initial window length. Represents the half wavelength corresponding to the periodic characteristics of the signal, where N is the length of the depth distance signal, and i∈[k*, Nk*-1]; S42: Detect the true peak value of the respiration change curve using the optimal window, and count the number of times each position satisfies the multi-scale maximum value, denoted as C(i). The expression for C(i) is as follows: In the formula, Rules for determining peak values.

[0015] Secondly, this invention discloses a non-contact RGB-D-based system for detecting the respiratory rate of recumbent cattle. The detection system is used to implement the aforementioned detection method and includes: The data acquisition module is used to acquire and preprocess RGB-D video stream data of lying cattle in a non-contact manner. The key area selection module uses the RGB visible light video stream in the preprocessed RGB-D video stream data to locate the key parts of the side-lying cattle and define the key target area of ​​the abdomen. The registration module is used to register the position coordinate correspondence between the RGB visible light video stream and the depth video stream; The depth calculation module is used to calculate the average depth value of the key target area on the abdomen after registration; The curve plotting module is used to plot respiratory change curves based on the average depth changes of key target areas in the abdomen. The peak detection module is used to detect peaks in the respiratory change curve by combining the AMPD algorithm, count the number of peaks, and obtain the number of breaths.

[0016] In addition, the present invention also discloses an electronic device, the electronic device including at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method described above.

[0017] Compared with the prior art, the present invention has the following advantages: 1. This invention proposes an RGB-D-based detection method for respiratory frequency in cattle resting on their sides. To optimize the selection of regions with obvious abdominal respiratory features, after reading the RGB visible light video stream from the RGB-D video stream data, the YOLOv8n-Pose network structure is used to mark key points on four key parts of the cattle's neck, forelimbs, and hindlimbs when the cattle are lying on their sides. Based on the geometric constraints of the coordinates, the key points of the cattle's abdominal breathing are dynamically calculated, and the key target regions of the abdomen are divided. Then, the depth video stream and the RGB visible light video stream are aligned frame by frame. The target regions in the RGB visible light image are mapped to the depth image to obtain the depth values ​​within the key target regions of the abdomen. Based on the depth changes, a respiratory waveform curve is established to realize the detection of respiratory frequency.

[0018] 2. The method of the present invention has a coefficient of determination (R²) of 0.94 for detecting the respiratory rate of laterally recumbent cattle compared with manual observation, an average detection accuracy of 93.3%, and a root mean square error of 3.75 times / minute. This not only confirms the rationality of the dynamic positioning strategy of the abdominal region, but also verifies the reliability of the RGB-D-based method for detecting the respiratory rate of laterally recumbent cattle, providing technical support for realizing non-contact intelligent detection of cattle respiratory rate. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0020] Figure 1 This is a flowchart illustrating the technical process of the non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to the present invention. Figure 2 Figure A shows the device for acquiring RGB-D video stream data of the abdomen of a cow lying on its side, according to the present invention. Figure 3This is a structural diagram of the Intel RealSense D435 depth camera of the present invention; Figure 4 This is a schematic diagram showing the key points of the side-lying cattle in this invention; Figure 5 This is a dataset of images of cows lying on their sides, as presented in this invention. Figure 6 For corresponding Figure 5 The annotation file for each image in the document; Figure 7 This is a schematic diagram illustrating the location of the key target area on the abdomen of a cow lying on its side, according to the present invention. Figure 8 A schematic diagram illustrating the occlusion of a key target area on the abdomen of a cow lying on its side, as per the present invention. Figure 9 The following is a comparison of depth histograms of key target areas on the abdomen of a lateral-lying cow, as presented in this invention: a) represents the unobstructed histogram, and b) represents the obstructed histogram. Figure 10 This is a diagram illustrating the effect of aligning the depth map of the present invention to an RGB visible light image; Figure 11 This is a flowchart illustrating the process of calculating the distance to key target areas on the abdomen of a cow in a lateral recumbent state, according to the present invention. Figure 12 This is a graph showing the peak detection effect of the depth change curve of the present invention; Figure 13 This is a diagram illustrating the positioning effect of the key target area on the abdomen of a cow lying on its side according to the present invention. Figure 14 The first example of the respiratory change curve and peak detection effect of the present invention is shown in the figure, where a is a randomly selected first 30-second respiratory detection video and b is a randomly selected second 30-second respiratory detection video. Figure 15 The second example of the respiratory change curve and peak detection effect of the present invention is shown in the figure, where c is the third 30-second respiratory detection video selected randomly, and d is the fourth 30-second respiratory detection video selected randomly. Figure 16 A schematic diagram showing the selection of the abdominal optical flow detection area of ​​a side-lying cattle according to the present invention; Figure 17 This is a comparison chart showing the correlation between the prediction results of the method of the present invention and the optical flow method; Figure 18 A figure showing the respiratory rate change curve and peak detection results obtained by using optical flow method to detect respiratory rate in a summer scene with sufficient sunlight. Figure 19 Two figures show the respiratory rate change curves and peak detection results obtained by using optical flow method to detect respiratory rate in a summer scene with sufficient sunlight. Figure 20A figure showing the respiratory rate change curve and peak detection results obtained by using optical flow method to detect respiratory rate in a winter scene with insufficient light; Figure 21 Two figures show the respiratory rate change curves and peak detection results obtained by using the optical flow method to detect respiratory rate in a winter scenario with insufficient light. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0022] Reference Figures 1-21 This invention provides a non-contact method for detecting the respiratory rate of recumbent cattle based on RGB-D, comprising the following steps: S1: Collect RGB-D video stream data of lying cattle in a non-contact manner, and preprocess the RGB visible light video stream and depth video stream in the RGB-D video stream data respectively; S2: Based on the preprocessed RGB visible light video stream, locate the key parts of the side-lying cattle, and combine the key parts to define the key target area of ​​the abdomen; S3: Based on the positional coordinate correspondence between the RGB visible light video stream and the depth video stream, registration is performed to obtain the average depth value of the key target area in the abdomen; S4: Plot the respiratory change curve based on the average depth change of the key target area in the abdomen, and use the AMPD algorithm to detect peaks in the respiratory change curve, count the number of peaks, and obtain the number of breaths.

[0023] Specifically, in step S1, an Intel RealSense D435 depth camera is connected to a tripod using 1 / 4-inch gimbal screws. The camera is 80 cm off the ground and is positioned at a 45° angle to the side and rear of the cattle's resting area to record an RGB-D video stream. The distance between the camera and the target cattle is 2.5 meters. The on-site data acquisition device is as follows: Figure 2 As shown, the camera is connected to the computer via a USB 3.2 cable. Video recording begins using the Intel RealSense SDK. The video data is saved in BAT file format. The image resolution of both batches of depth videos is 848×480, and the video data frame rate is 60 FPS. The camera structure of the Intel RealSense D435 depth camera is as follows: Figure 3As shown, the camera includes an RGB camera, two infrared cameras, and an infrared transmitter. The RGB resolution is up to 1920×1080, and the infrared cameras have a binocular depth resolution of up to 1280×720. The depth camera, Intel RealSense D435, uses binocular vision to measure depth distance and reads the acquired RGB-D depth video stream of the lying cow. It can simultaneously acquire visible light RGB video stream and depth stream data.

[0024] The collected RGB-D video data of the cows' resting lateral recumbent areas was saved in BAT format. The data collection subjects were 32 lactating cows. The lateral recumbent depth data of 24 cows was collected under sufficient lighting conditions, and that of 8 cows was collected under low light conditions. The BAT video files of these 32 cows underwent sequence frame parsing processing at a frame rate of 60 FPS, resulting in a total of 112 video segments. These segments contained 1540 images, including RGB visible light images and depth maps with depth distance information. The Labelme annotation tool was used to create matrix bounding boxes for each cow and to annotate four key body parts. The file format was JSON, and the annotations are as follows: Figure 4 As shown, to adapt to the YOLOv8n-Pose network structure used in this invention, JSON tags are converted into YOLO format txt files, such as... Figures 5-6 As shown.

[0025] In step S2, to detect the respiratory rate of cattle in a side-lying resting state, the change in abdominal undulation distance is determined based on the cattle's side-lying posture. When cattle are resting on their sides, their forelimbs are curled under their bodies or extended forward, one hind leg extends forward under their abdomen, and most of their weight is supported by the triangular structure formed by the ischial tuberosity, knee joint, and tarsal joint. The other hind leg extends to the side of the body, with the knee and tarsal joints partially flexed. There are two body positions: left-side or right-side lying. Based on the cattle's side-lying posture, the visible light image in the RGB-D depth video stream is read, and the keypoints of the side-lying cattle are located using the YOLOv8n-Pose network structure. The YOLOv8n-Pose network structure is a keypoint localization model extended from the YOLOv8 object detection framework, adding keypoints to the original detection head. The Head branch and the parallel prediction branch retain the object detection function while also having the key point localization function. This avoids the problem of directly using object detection boxes to define the abdominal area of ​​the lying cow, which has poor detection effect and serious randomization of abdominal area acquisition. In addition, there will be slight motion interference during the lying process of the cow, which can easily introduce noise in the abdominal depth value.

[0026] To accurately capture the abdominal region of cattle, four key areas were marked and located on the side-lying cattle: the base of the neck, the knee joint of the foreleg, the knee joint of the hind leg, and the base of the hind leg. To address the issues of obstruction by the bedding railings and poor localization in side-lying cattle, the following key points were selected: the base of the neck (point A), the knee joint of the foreleg (point B), the base of the hind leg (point C), and the knee joint of the hind leg (point D). The intersection of the lines connecting points A and D, and B and C, was determined, and a rectangular area centered on this intersection was captured. This abdominal region was then used as the key target area for detection, and subsequent depth value changes were recorded. The localization method for the entire key target area of ​​the abdomen is as follows: Figure 7 As shown.

[0027] Specifically, the direction vector of line segment AD connecting the two points is denoted as (x2-x1, y2-y1), and the direction vector of line segment BC connecting the two points is denoted as (x4-x3, y4-y3). To determine whether the two direction vectors intersect, equation (1-1) is used to represent the relationship between the two direction vectors. These two vectors are considered as a matrix determinant. D1 is used to determine whether the two vectors are parallel or coincident. When D1 is 0, it indicates that the direction vectors of the two line segments are linearly related and there is no intersection. After checking whether the two line segments are parallel, the intersection of the two line segments is determined, as shown in equations (1-2) and (1-3). The parameters t and s indicate the position of the intersection point on line segment AD and line segment BC, respectively. When the values ​​of t and s are in the range [0,1], it means that the intersection point is on both line segment AD and line segment BC, that is, the intersection of the two line segments exists. Finally, the coordinates (x, y) of the intersection point are calculated using equation (1-4). Based on the coordinates of the intersection point, a rectangular area of ​​a certain range is extracted.

[0028] When dynamically selecting the abdominal examination area based on the key point localization results of the cattle's neck and fore and hind limbs, the designated area may be obscured by the bedding area's railings, such as... Figure 8As shown, the dynamically defined abdominal region is obscured by a bed rail, resulting in fewer effective depth values ​​related to breathing and a weaker effective respiratory signal, affecting the accuracy of respiratory rate detection. To determine whether the currently defined 40px × 40px rectangular region is obscured by a bed rail, depth values ​​acquired by a depth camera within the rectangle are analyzed to determine the distribution pattern. In normal, unobstructed areas, depth differences are small. When the depth value difference within a rectangular region is large, it indicates that the abdominal region within the rectangle is obstructed. To determine the depth difference within the selected abdominal region, the histogram distribution within the region is analyzed, and the variance of the histogram is calculated. In unobstructed areas, depth values ​​show a concentrated distribution with small variance; when obstructed by a bed rail, the histogram shows a multi-peaked distribution and a significantly increased variance. Figure 9 As shown, a variance threshold is set to determine whether there is occlusion.

[0029] If it is determined that the current key target area of ​​the abdomen is obstructed, the rectangular target area of ​​the abdomen is adjusted to accurately obtain the area containing effective respiratory features. Combined with the side-lying resting state of the cattle, when obstruction occurs, a sliding window is set in the area enclosed by four key points based on the initially selected area. The window is slid down 30 pixels to readjust the target area, keeping the size of the target area the same. The abdominal respiratory signal of the cattle is perceived based on the change in the average depth value of the readjusted area.

[0030] In this invention, the registration of RGB visible light images and depth maps is completed based on the coordinate system transformation relationship between depth maps and color maps. By transforming the coordinates, the coordinate information of the visible light image is synchronously mapped to the depth image frame of the same frame to obtain the depth value of the key target area of ​​the abdomen at the same location, so as to achieve the purpose of obtaining the change of abdominal depth value by means of the semantic information features of visible light.

[0031] The depth image and color image were acquired by the depth camera and RGB camera modules of the D435 depth camera, respectively. The depth camera intrinsic parameter K was obtained from the software development kit provided by the Intel RealSense camera. d The intrinsic parameters K of the RGB camera c The transformation matrix T from the world coordinate system to the depth camera coordinate system is obtained through calibration. w2d Transformation matrix T from world coordinate system to color camera coordinate system w2c And the transformation matrix T from the depth camera coordinate system to the color camera coordinate system. d2c .

[0032] To transform pixels in the depth image coordinate system to the color image coordinate system, the pixels in the depth image are first restored to the depth image coordinate system. Then, the coordinates in the depth coordinate system are restored to the world coordinate system. Next, the depth points in the world coordinate system are transformed to the color coordinate system. Finally, they are mapped onto the Z-axis normalized plane and mapped to correspond with the pixels in the color image. The entire transformation process is derived from equations (1-5) to (1-9), where: In the formula, Represents the coordinates of a point in the coordinate system of a depth image. Z represents a pixel in the depth image, where Z represents the depth value of that pixel. This represents the intrinsic parameter matrix of the depth camera; Pixels in the depth map After restoration, the depth point is... Transform to the world coordinate system to obtain the spatial points in the color camera coordinate system. Transforming the coordinate system of the depth camera to that of the color camera requires a Euclidean transformation matrix as a transition. The form of the Euclidean transformation matrix T is shown in equation (1-7). Finally, the Euclidean transformation matrix is ​​used... As shown in equation (1-8), the coordinate points in the depth camera spatial coordinate system can be directly transferred to the color camera spatial coordinate system. In equation (1-9), the depth points in the color camera coordinate system are... Mapping onto the color plane with Z=1, and normalizing the Z-axis to complete the planar mapping: In the formula, The depth point is represented in the coordinate system of the color image, and T is the Euclidean transformation matrix. This is the transformation matrix for converting from the world coordinate system to the color camera coordinate system. R is the transformation matrix from the coordinate system of the depth image to the world coordinate system, R is the rotation matrix, is unit orthogonal, and t is the translation amount of the axis.

[0033] After the above conversion process, combined with the depth map and camera parameters, the task of aligning the depth image to the RGB visible light image was completed. The aligned result is shown in the image below. Figure 10 As shown.

[0034] After obtaining the depth map of the key area of ​​the abdominal target, the distance value represented by the depth information is calculated, such as... Figure 11 As shown, the ratio α between the depth image pixel size and the actual physical distance is read from the depth camera to obtain the key target area of ​​the side-lying cow's abdomen. A rectangle is returned, and the pixel value is obtained by traversing all pixels within the rectangle. Based on the ratio of the pixel value to the physical distance, the physical distance represented by each pixel is calculated and recorded. The distance of the entire selected key target area of ​​the abdomen is averaged to reduce the influence of sensor noise. Finally, the average distance value of the key target area of ​​the abdomen in the current image frame is obtained. The video stream is read continuously, and the distance value of each frame is obtained.

[0035] Furthermore, respiratory feature signals are extracted based on the changes in depth distance values ​​of the target region in each frame of the RGB-D video stream, ultimately completing the detection of respiratory frequency. Specifically, an abdominal depth value change curve is plotted based on continuous abdominal region depth distance values ​​for determining respiratory status. When cattle are resting in a side-lying position, exhaling causes the abdomen to concave inward, increasing the distance between the abdominal detection area and the depth sensor, resulting in an upward trend in the depth distance change curve. Conversely, inhaling causes the abdomen to bulge outward, decreasing the distance between the abdominal detection area and the depth sensor, resulting in a downward trend in the plotted abdominal distance change curve. After plotting the depth distance change curve, during the processing of this digital signal, the number of peaks is accumulated as the number of breaths for counting. Therefore, it is necessary to detect, identify, and count the peaks of the change curve. This invention utilizes the Automatic Multi-Scale Peak Finding Algorithm (AMPD) to find peaks in the respiratory cycle signal.

[0036] The AMPD algorithm compares the values ​​of the indices on both sides of a multi-scale sliding window to find the maximum value within a local window. After determining the optimal sliding detection window size, it performs true peak detection. In the first step of window size determination, the calculation method is shown in equations (1-10) and (1-11), initializing the window length k and the length of the depth distance signal as N. During window traversal, the length k ∈ S(k) is the weight of the local maxima within each window, and the selection of the optimal window requires minimizing S(k).

[0037] In the formula, S(k) represents the local maximum weight within each window, and k is the initial window length. Represents the half wavelength corresponding to the periodic characteristics of the signal, where N is the length of the depth distance signal, and i∈[k*, Nk*-1]; This is an indicator function; its value is 1 if the condition is met, and 0 otherwise. i Let k be the i-th sampling point of the continuous signal, and k be the window length. * This is the half-wavelength corresponding to the periodic characteristics of the signal.

[0038] After determining the optimal window size, the true peak value of the distance change curve is detected based on the optimal window. For each position index i∈[k*,Nk*-1], the number of times the multi-scale maximum value is satisfied is denoted as C(i), which is used to measure the scale consistency of the point as a local maximum value. The method for calculating the cumulative number of times C(i) is satisfied at the position is shown in Equation (1-12). Finally, the peak value determination rule is shown in Equation (1-13).

[0039] In the formula, Rules for determining peak values.

[0040] Based on the peak detection AMPD algorithm, the number of breaths is counted on the depth-distance change curve, and the detection effect is as follows: Figure 12 As shown, the horizontal axis represents the index subscript in each frame, and the vertical axis represents the distance between the abdominal target area and the depth camera, in meters. The figure shows the peak detection effect of the distance change curve over 30 seconds. Combined with the detection duration of the video, the respiratory rate per minute is obtained according to the respiratory rate detection formula, thereby completing the detection of the respiratory rate of the side-lying cattle.

[0041] The YOLOv8n-Pose network structure used in this invention is an extension of the YOLOv8 object detection series. A keypoint detection branch is added at the final detection head of the model. Therefore, evaluating the keypoint localization performance of this model when cattle are lying on their sides includes evaluating the object detection performance of the entire cattle as well as the localization performance of keypoints in different parts of the cattle. Therefore, this invention evaluates the localization performance based on four indicators: Precision (detection accuracy of the detection box), Recall, Average Precision (mAP50) at an IoU threshold of 0.5, and Average Precision (mAP50-95) from an IoU threshold of 0.5 to 0.95. The average precision of the predicted boxes is denoted as mAP50(Box), and the average precision of the keypoints is denoted as mAP50(Pose).

[0042] This invention focuses on the effectiveness of respiratory rate detection in laterally recumbent cattle, selecting Precision, Root Mean Square Error (RMSE), and Coefficient of Determination (R²) as the parameters. 2 These three evaluation indicators comprehensively assess the effectiveness of a detection method based on the degree of matching between the detected value and the true value, the true nature of the error distribution, and the stability of the method. The specific calculation methods for these three evaluation indicators are shown in the following formulas:

[0043] Example:

[0044] The respiratory rate data of the lateral recumbent cattle used in this embodiment were collected from abdominal depth data in August 2023 at Shengkang Livestock Farm in Lindian County, Daqing City, Heilongjiang Province, and in December 2024 at Furui Livestock Farm in Shuangcheng District, Harbin City, Heilongjiang Province.

[0045] 1) Key point detection results of cattle lying on their sides: After processing the collected RGB-D video stream data of lying cattle, it was divided into training and testing sets in an 8:2 ratio. The training set images were then fed into the YOLOv8n-Pose network for training. The training parameters are shown in Table 1. After 200 iterations, the best-performing PyTorch model was saved locally for future applications. The final evaluation metrics of the trained YOLOv8n-Pose network model are shown in Table 2. The RGB visible light images showed a small number of cattle and a single target, with clear location features for the four key points of the cattle. The average accuracy of the predicted bounding boxes and key points was 96.1% and 99.5%, respectively. The overall localization effect and the selection of the abdominal region were as follows: Figure 13 As shown.

[0046] Table 1 Training parameter configuration of YOLOv8n-Pose network structure

[0047] Table 2. Target detection and key point localization performance of the YOLOv8n-Pose network structure

[0048] Depend on Figure 13 The four positioning results show that the cattle's lateral recumbent postures are divided into left lateral and right lateral recumbent positions. Different individual cattle have different postures when lying on their sides, which requires high robustness of the model for positioning the cattle's neck root and fore and hind limbs. Dynamically positioning a rectangular area based on key points on the abdomen is mainly to solve the problem of normal body swaying of cattle interfering with the normal detection process during respiratory rate detection.

[0049] 2) Selection of the key target area on the abdomen of a cow lying on its side: Based on the strategy of locating the abdominal region using four key points, after determining the abdominal location points, the size of the delineated area of ​​the key target region needs further selection. The average depth value of the delineated area is calculated using RGB-D data to perceive the animal's respiratory state. To determine the specific size of the selected area for the abdominal region of a side-lying cattle, this embodiment divides a rectangular region centered on the located key point of the abdominal region. To determine the optimal region selection size, the resolution of the captured local image is first set to H×H, with H values ​​set to 20, 30, 40, and 50 respectively. The average depth value within these regions is extracted, and the final local image resolution is selected based on the final respiratory rate detection accuracy.

[0050] surface Comparison of detection effects at different resolutions

[0051] The results presented in Table 3, based on the four different resolution metrics, show that a local image resolution of 40×40 yields the best respiratory rate detection performance. Analysis of the various metrics in Table 3 reveals that when the resolution is too low, the weak variation in the mean depth of the abdominal region prevents the extraction of effective depth information from the abdominal respiratory area, resulting in the partial loss of key respiratory features and ultimately leading to poor model detection performance. When the image resolution is too high, the extracted abdominal area is too large, and in some data, this area even includes the hind limbs. Excessive noise introduction reduces the accuracy of respiratory rate detection, resulting in a larger detection error.

[0052] Therefore, by comparing and analyzing the detection performance of respiratory rate at different local image resolutions, the optimal resolution for the key abdominal target region was determined to be 40 pixels × 40 pixels. This optimal region selection not only significantly improved the accuracy and stability of respiratory rate detection but also effectively reduced interference from background noise and non-respiratory movements.

[0053] 3) Results of respiratory rate detection in cattle lying on their sides: To evaluate the effectiveness of detecting changes in the abdominal depth of cattle based on RGB-D video streams, the respiratory rate detection results were compared with the actual respiratory rate detection results. For the actual values ​​of the RGB-D respiratory rate detection data, the collected bag-format data was read using Intel RealSense's Viewer visualization software. Respiratory fluctuations were counted manually, with one complete rise and fall of the abdomen of a side-lying cattle counted as one breath. For the same video segment, the observation was repeated three times manually, and the average of the three observations was taken as the actual value, thus ensuring the reliability of the actual data.

[0054] The test and validation dataset for abdominal respiratory rate detection consists of 82 lateral recumbent videos, each 30 seconds long. The model predictions for these 82 videos are compared with the actual results. The final detection performance is as follows: Figures 14-15 As shown in Table 4, the respiratory rate detection results of this method are as follows: the average detection accuracy of the model is 93.3%, the root mean square error between the automatic detection results and the actual results is controlled at 3.75 breaths / minute, and the coefficient of determination R0 is [value missing]. 2 It is 0.94.

[0055] Table 4. Results of respiratory rate detection in cattle lying on their sides

[0056] The dataset used in this embodiment consists of RGB-D video stream data collected in summer and winter. The cattle in the barn are in different environments. Analysis of the summer data revealed that when cattle are lying on their sides, they exhibit head shaking, tail wagging, and skin twitching. Summer barns are hot and humid with many mosquitoes and flies. To repel these insects, the cattle perform these actions. For example, sudden body twitching causes changes in the distance between the abdomen and the depth acquisition device, resulting in abnormal fluctuations in the depth value curve, negatively impacting the accuracy of respiratory rate detection. Simultaneously, from Figure 13 As shown, when tracking the abdominal region from the side and rear of the cattle, there are instances where the abdominal area is partially obscured by the bedding. By employing a strategy of dynamic localization of multiple key points on key body parts of the cattle, this situation of being unable to detect breathing due to obstruction is avoided. The final breathing signal is captured by sensing changes in the depth value of the unobstructed area within a rectangular bounding box. Through a strategy of dynamic extraction of the rectangular region, the breathing signal is captured to the maximum extent, ultimately achieving automatic detection of the cattle's respiratory rate.

[0057] 4) Comparison and analysis of this invention with the RGB optical flow detection method Existing technologies primarily rely on optical flow detection in visible light images for respiratory rate detection. This involves sensing respiratory signals and counting respiratory counts based on the average optical flow direction change along the abdominal mottled border. In this embodiment, the abdominal depth information extraction region is dynamically located by key points from other areas and bounded by a rectangle. The depth value of each pixel within the rectangle is extracted, and the average depth value within the region is selected as the depth value in the image at the current moment. Based on the coordinates of the located rectangle, the abdominal region in the RGB visible light video stream is dynamically cropped and saved. The selected abdominal rectangular image has a resolution of 40×40. The abdominal region selection in the RGB visible light image is as follows: Figure 16 As shown.

[0058] To compare the performance with optical flow detection methods, 40 depth videos were randomly selected from all RGB-D depth videos for optical flow variation extraction. Twenty videos were collected in summer, and the other 20 were collected in winter. From these 40 selected videos, the accuracy, RMSE, and R... 2 The indicators were compared and evaluated, and the results are shown in Table 5.

[0059] Table 5 Comparison of prediction results between the method of the present invention and the optical flow detection method The quantitative analysis and comparison results in Table 5 show that the respiratory rate detection effect based on optical flow is weaker than that of the method of the present invention.

[0060] On the one hand, in the RGB-D video stream data of cattle lying on their sides in summer, the cattle are resting in the resting area. In summer, mosquitoes in the cattle shed disturb the individual cattle, which not only affects the comfort of the cattle and reduces the yield of individual cattle, but also causes the cattle to exhibit stress responses such as head shaking, tail wagging, and skin twitching in order to expel the insects. The presence of this external mosquito interference factor has a certain negative impact on the detection results of the cattle's respiratory rate. When using the RGB visible light optical flow method to sense the respiratory signal, the cattle's insect-expelling actions will cause abnormal changes in the optical flow field, resulting in deviations in the counting of respiratory counts, and ultimately affecting the detection of respiratory rate.

[0061] On the other hand, during the entire breathing process of a cow, the abdomen undergoes a certain horizontal displacement. This can be detected by comparing pixel changes between two consecutive frames using the LK sparse optical flow method. For example... Figure 17As shown, the correlation between the respiratory rate detected by optical flow and the true value is slightly lower than that of the respiratory rate detected based on depth change in this invention. Analysis of the compared datasets shows that the RGB-D video stream data of the lying cattle consists of two parts: a scene with ample sunlight in summer and a scene in dim light inside the shed in winter. Based on the fundamental principle of optical flow for detecting pixel motion, the optical flow method can fully function under the premise of constant brightness and small-amplitude movement. Figures 18-21 As shown, when there is sufficient sunlight in summer, the directional changes in the optical flow field can be accurately captured, the breathing curve is smooth and has obvious peak characteristics, and the counting of breathing frequency can be accurately completed. However, when the indoor environment is dim in winter, the quality of abdominal feature points degrades, and abdominal texture information is difficult to capture, thus affecting the stability of the sparse optical flow algorithm. The method of this invention captures the changes in the depth distance value of the abdomen, fits a depth change curve based on the depth mean of the rectangular region of the abdomen, and utilizes the depth sensing function of the depth sensor to avoid the impact of low light on the effectiveness of respiratory rate detection. At the same time, even when the abdominal region is obstructed by the bed rails, the changes in the depth value of the rectangular region of the abdomen can still be accurately captured, thereby completing the calculation of respiratory rate.

[0062] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A non-contact method for detecting the respiratory rate of recumbent cattle based on RGB-D, characterized in that, Includes the following steps: S1: Collect RGB-D video stream data of lying cattle in a non-contact manner, and preprocess the RGB visible light video stream and depth video stream in the RGB-D video stream data respectively; S2: Based on the preprocessed RGB visible light video stream, locate the key parts of the side-lying cattle, and combine the key parts to define the key target area of ​​the abdomen; S3: Based on the positional coordinate correspondence between the RGB visible light video stream and the depth video stream, registration is performed to obtain the average depth value of the key target area in the abdomen; S4: Plot the respiratory change curve based on the average depth change of the key target area in the abdomen, and use the AMPD algorithm to detect peaks in the respiratory change curve, count the number of peaks, and obtain the number of breaths.

2. The non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to claim 1, characterized in that, In step S2, the specific steps for locating the key areas of the lying-on-side cattle and defining the key target area of ​​the abdomen based on these key areas include: S21: Using the YOLOv8n-Pose network structure, four key points were located on the neck and forelimbs of the side-lying cattle. S22: Based on the positions of four key points, dynamically select the coordinate information of the abdomen of the lying cow to obtain the key target area of ​​the abdomen.

3. The non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to claim 2, characterized in that, Step S22, the steps for obtaining the key target region of the abdomen, include: Connect the two opposite corners of the four key points; Determine the intersection of the two lines, obtain the rectangular area centered on the intersection, and use the rectangular area as the key target area of ​​the abdomen.

4. The non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to claim 3, characterized in that, Before step S3, if it is determined that the current key abdominal target area is occluded, a sliding window is set for the key abdominal target area enclosed by the four currently selected key points, and a new key abdominal target area is obtained by adjusting the rectangular area.

5. The non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to claim 3, characterized in that, Step S3, which involves obtaining the average depth value of the key target region on the abdomen, includes: S31: Acquire an RGB visible light image and a depth image containing depth and distance information; S32: Based on the coordinate system transformation between the RGB visible light image and the depth image, registration is performed to obtain the depth distance value between the key target area on the abdomen in the RGB visible light image and the same coordinate position in the depth image.

6. The non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to claim 3, characterized in that, Step S32, the registration step based on the coordinate system transformation between the RGB visible light image and the depth image, includes: The expression for restoring the pixels of a depth image to the coordinate system of the depth image, and restoring the coordinates of the points in the coordinate system of the depth image to the world coordinate system, is as follows: In the formula, Represents the coordinates of a point in the coordinate system of a depth image. Z represents the pixel in the depth image, and Z represents the depth value of the pixel. This represents the intrinsic parameter matrix of the depth camera; The coordinates of points in the world coordinate system are transformed to the coordinate system of the color image using a Euclidean transformation matrix. The expression is as follows: In the formula, The depth point is represented in the coordinate system of the color image, and T is the Euclidean transformation matrix. This is the transformation matrix for converting from the world coordinate system to the color camera coordinate system. R is the transformation matrix from the coordinate system of the depth image to the world coordinate system, R is the rotation matrix, is unit orthogonal, and t is the translation amount of the axis. The depth points in the coordinate system of the color image are mapped to the color image plane with Z=1, and the Z-axis is normalized to complete the task of aligning the depth image to the RGB visible light image. The expression is as follows: In the formula, This indicates the pixel coordinates of the depth point on the RGB visible light image. This indicates the intrinsic parameters of the RGB camera.

7. The non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to claim 6, characterized in that, The steps to obtain the average depth value of the key target region in the abdomen include: Obtain the ratio α of the pixel values ​​of the depth image to the physical distance; Traverse all pixels in the key target area of ​​the abdomen and calculate the distance value represented by each pixel according to the ratio α; The average distance value of the key target region in the abdomen is calculated based on the distance value represented by each pixel.

8. The non-contact lateral recumbent cattle respiratory rate detection method based on RGB-D according to claim 7, characterized in that, Step S4, which involves using the AMPD algorithm to detect peak values ​​in the respiration change curve, includes: S41: The sliding detection window is determined based on the AMPD algorithm. The calculation formula for confirming the window size is as follows: In the formula, S(k) represents the local maximum weight within each window, and k is the initial window length. Represents the half wavelength corresponding to the periodic characteristics of the signal, where N is the length of the depth distance signal, and i∈[k*, Nk*-1]; S42: Detect the true peak value of the respiration change curve using the optimal window, and count the number of times each position satisfies the multi-scale maximum value, denoted as C(i). The expression for C(i) is as follows: In the formula, The rules for determining peak values.

9. A non-contact RGB-D-based system for detecting the respiratory rate of recumbent cattle, characterized in that, The detection system is used to implement the detection method according to any one of claims 1-8, and the detection system comprises: The data acquisition module is used to acquire and preprocess RGB-D video stream data of lying cattle in a non-contact manner. The key area selection module uses the RGB visible light video stream in the preprocessed RGB-D video stream data to locate the key parts of the side-lying cattle and define the key target area of ​​the abdomen. The registration module is used to register the position coordinate correspondence between the RGB visible light video stream and the depth video stream; The depth calculation module is used to calculate the average depth value of the key target area on the abdomen after registration; The curve plotting module is used to plot respiratory change curves based on the average depth changes of key target areas in the abdomen. The peak detection module is used to detect peaks in the respiratory change curve by combining the AMPD algorithm, count the number of peaks, and obtain the number of breaths.

10. An electronic device, characterized in that: The electronic device includes at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor to enable the processor to perform the method of any one of claims 1-8.

Images