Rapid live pig body temperature detection method based on image registration and multivariate information fusion

Abstract

The invention discloses a live pig body temperature rapid detection method based on image registration and multivariate information fusion, and relates to the technical field of animal husbandry facilities and detection. The method comprises the following steps: (1) setting up a device (S1); (2) image registration (S2); (S3) data acquisition; (S4) key point detection; (5) model training (S5); (6) deploying the model (S6); 7, body temperature detection (S7). According to the method, the infrared thermal image and the RGB image are registered, so that the imaging error caused by a single imaging source is prevented, and the details of the key area of the live pig to be detected are effectively improved; the robustness and the accuracy of infrared temperature detection are improved by acquiring the environment information, the temperature information of the key area and the temperature difference and inverting the real body temperature of the live pig by using the multi-layer perceptron neural network.

Description

Technical Field

[0001] The present invention relates to the field of animal husbandry facilities and detection technology, and in particular to a method for rapid detection of pig body temperature based on image registration and multivariate information fusion. Background Art

[0002] With the development of modern breeding technology and information technology, the traditional pig breeding model is changing to large-scale, automated and welfare-oriented breeding; at the same time, the problems of deteriorating breeding environment and disease outbreaks are becoming increasingly prominent. Pig breeding is a link that farms pay more attention to. Body temperature is a basic indicator for judging the health of pigs. Factors such as disease or physiological changes can cause the body temperature of pigs to exceed the normal range. In addition, among many infectious disease symptoms of pigs, the increase in body temperature appears earlier than other symptoms. Monitoring and analyzing the body temperature of pigs is conducive to timely grasping the physiological health status of pigs, timely discovering physiological abnormalities of pigs, so that breeders can take corresponding measures in time, ensure the health level of pigs, improve animal welfare, and reduce economic losses caused by pig diseases.

[0003] At present, with the development of precision animal husbandry, infrared thermal imagers are applied to livestock and poultry farming for the detection of animal fever, pain, edema, lameness caused by inflammation, estrus and ovulation detection, etc. The main detection idea of ​​infrared thermal imaging technology is to use image recognition to detect the surface temperature of key positions of livestock and poultry, and use the surface temperature to approximate the real body temperature of livestock and poultry. However, infrared thermal imaging is easily affected by environmental factors, resulting in low detection accuracy and is not suitable for commercial detection. Using image recognition technology to detect infrared thermal images, because the details of infrared thermal images are blurred and the existing detection methods have high requirements for image details, simply using heat maps for image recognition often does not achieve satisfactory results. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, such as blurred imaging details of infrared thermal images and susceptibility to environmental factors resulting in poor temperature detection accuracy. A method for rapid detection of pig body temperature based on image registration and multi-information fusion is provided, which registers RGB images with IR images, eliminates the shortcomings of blurred details of infrared images, and uses multi-information fusion modeling to effectively improve the accuracy of simple infrared temperature measurement.

[0005] The object of the present invention is achieved in that:

[0006] By combining the IR camera and the RGB camera, the two different types of images are matched using image registration technology, and the key parts of the pigs in the RGB image are accurately located through the deep learning target detection algorithm and mapped to the temperature information matrix, reducing the temperature information error interference caused by a single image. Since the temperature information obtained by the infrared thermal imager is easily interfered by environmental factors, the environmental information obtained by the environmental temperature and humidity sensor is combined to comprehensively calculate the real temperature of the pig using a deep learning model.

[0007] Specifically, the method comprises the following steps:

[0008] ①Device construction: Build a pig image information collection system and an environmental information collection system. The image information collection system includes an IR camera and an RGB camera, and the environmental information collection system includes an environmental temperature and humidity sensor, a wind speed sensor, and a light intensity sensor;

[0009] ② Image registration: fix the relative position and shooting distance of the IR camera and the RGB camera, use the algorithm to calculate the mapping relationship between the two images, and complete the image registration;

[0010] ③Data collection: The device is installed at the front end of the limiting fence to collect infrared thermal images, RGB images and environmental information of pig faces;

[0011] ④Key point detection: Use the improved YOLO target detection algorithm to detect the eyes, ear roots, forehead and nose positions of the pigs in the RGB image, and obtain the temperature information of the key points through the mapping relationship;

[0012] ⑤Model training: The key point temperature information and environmental information obtained are made into a data set for deep learning model training to obtain a real-time pig body temperature detection model;

[0013] ⑥Model deployment: Deploy the model to the Jetson Orin NX development board to achieve real-time inference of the model on the embedded device;

[0014] ⑦Body temperature detection: Install the device at the front end of the limit fence to detect the body temperature of live pigs.

[0015] The present invention has the following advantages and positive effects:

[0016] ① By registering the IR image with the RGB image, the accuracy of detecting key areas of live pigs is increased, and detail errors caused by infrared images are prevented;

[0017] ②Use the key point temperature, ambient temperature and humidity, wind speed and light intensity information obtained by this device to comprehensively correct the infrared thermal imaging temperature measurement error;

[0018] ③ The YOLO target detection algorithm and the multi-layer perceptron neural network inversion model are used to obtain the real body temperature information of live pigs, which meets the production requirements of non-contact, stress-free and high accuracy in pig farms and improves the detection efficiency.

[0019] In summary, the present invention prevents imaging errors caused by a single imaging source by aligning the infrared thermal map with the RGB image, effectively improving the details of the key areas of the pigs to be detected; by acquiring environmental information, temperature information of key areas and temperature differences, and using a multi-layer perceptron neural network to invert the real body temperature of the pigs, the robustness and accuracy of infrared temperature detection are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a step diagram of this method, in which:

[0021] S1, S2...S7—steps ①②...⑦;

[0022] Figure 2 It is a schematic diagram of the structure of the device, in which:

[0023] 1—Display,

[0024] 2—Jetson Orin NX development board,

[0025] 3—IR and RGB cameras,

[0026] 4—wind speed sensor,

[0027] 5—Light intensity sensor,

[0028] 6—Limiting fence,

[0029] 7 - Pigs,

[0030] 8—Temperature and humidity sensor;

[0031] Figure 3 This is the network structure diagram of the temperature detection model;

[0032] Figure 4 It is the work flow chart of the present invention.

[0033] Abbreviations

[0034] 1. IR: Infrared Thermal (IRT) is short for infrared thermal imaging. Infrared thermal imaging uses photoelectric technology to detect infrared signals of specific wavelengths of thermal radiation from objects, converts the signals into images and graphics that can be distinguished by human vision, and can further calculate the temperature value. Infrared thermal imaging technology enables humans to overcome visual obstacles, so that people can "see" the temperature distribution on the surface of objects.

[0035] 2. RGB: refers to the three primary optical colors: R is red, G is green, and B is blue.

[0036] 3. Jetson Orin NX: artificial intelligence development board.

[0037] 4. PIN: Pin, also called tube pin, is called Pin in English. It is the connection between the internal circuit of the integrated circuit (chip) and the peripheral circuit. All the pins constitute the interface of this chip. The end of the lead wire is soldered to form a solder joint with the pad on the printed circuit board. The pin can be divided into the heel (bottom), toe (toe), side (side) and other parts. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The following is a detailed description with reference to the accompanying drawings and embodiments:

[0040] 1. Device

[0041] 1. Overall

[0042] like Figure 2 , the device includes a working object - a pig sample 7;

[0043] A display 1, a Jetson Orin NX development board 2, an infrared and RGB camera 3, a wind speed sensor 4, a light intensity sensor 5, a limit fence 6, and a temperature and humidity sensor 8 are provided;

[0044] Its location and connection relationship are:

[0045] The live pig sample 7 is in a limiting fence 6, which is equipped with an infrared and RGB camera 3, a wind speed sensor 4, a light intensity sensor 5, and a temperature and humidity sensor 8. The Jetson Orin NX development board 2 is connected to the infrared and RGB camera 3, the wind speed sensor 4, the light intensity sensor 5, and the temperature and humidity sensor 8, and is also connected to an external display 1.

[0046] 2. Functional components

[0047] The above-mentioned functional components are all universal parts.

[0048] Specific selection (model, specifications, etc.)

[0049] 1) Display 1

[0050] Selected: Pansheng E2434I-T 23.8-inch 100Hz FHD full HD IPS three-sided micro-edge crosshair display.

[0051] 2) Jetson Orin NX Development Board 2,

[0052] Selection: Jetson Orin NX 16GB, the processor is 8 ARM Cortex-A78AE, 16GB LPDDR5 high-speed memory, providing 1.5TFLOPS AI performance.

[0053] 3) Infrared and RGB cameras 3,

[0054] Selection: Tiny1-BE uncooled infrared thermal imaging module, measuring range -15℃~150℃, measuring accuracy ±2%, image resolution 256×192 pixels. USB visible light imaging module, image resolution 640×480 pixels.

[0055] 4) Wind speed sensor 4,

[0056] Selection: Thermal sensitive 485 anemometer, measuring range 1 ~ 15m / s, measuring accuracy ± 5%.

[0057] 5) Light intensity sensor 5,

[0058] Selection: TSL25911FN light meter, measuring range 0 ~ 88000Lux, sensitivity 188uLux.

[0059] 6) Limit fence 6

[0060] Selected: Nongfuteng livestock and poultry intelligent weighing equipment.

[0061] 7) Pig sample 7

[0062] Selection: French dual-purpose sows were used as research subjects.

[0063] 8) Temperature and humidity sensor 8

[0064] Selection: BME280 thermometer and hygrometer, temperature measurement range -40 ~ 85 ℃ (resolution 0.01 ℃, accuracy ± 1 ℃), relative humidity measurement range 0 ~ 100%, measurement accuracy ± 3%.

[0065] 3. Its working mechanism is:

[0066] Start the Jetson Orin NX development board 2, first test whether the infrared and RGB cameras 3, wind speed sensor 4, light intensity sensor 5 and temperature and humidity sensor 8 can work normally, then drive the pig 7 to the limit fence 6, and the RGB camera 3 starts to detect the facial image of the pig 7. After detecting a clear image, it takes an infrared thermal image and an RGB image, and tells the Jetson Orin NX development board 2 to send signals to the wind speed sensor 4, light intensity sensor 5 and temperature and humidity sensor 8 to obtain the corresponding environmental information, and obtain the real temperature of the pig through the temperature inversion model.

[0067] 2. Methods

[0068] like Figure 1 , the method comprises the following steps:

[0069] Step ①, device construction S1:

[0070] The IR camera module is combined with the RGB camera module to form an infrared and RGB camera 3 and connected to the Jetson Orin NX development board 2. The camera is controlled to take pictures through the corresponding program on the development board, and the ambient temperature and humidity sensor 8, wind speed sensor 4 and light intensity sensor 5 are connected through 40 pins of the development board to realize data transmission.

[0071] Step ②, image registration S2:

[0072] Before data collection, the infrared thermal imaging module and the RGB camera module need to be calibrated. The Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix K and the extrinsic parameter matrix M of the infrared thermal imaging module and the RGB camera module. (f x , f y Respectively represent the length of the focal length on the x-axis and y-axis, in pixels; u o With v o Represents the coordinates of the center point of the camera photosensitive plate in the pixel coordinate system); external parameter matrix (R is the rotation matrix and T is the translation vector), transform the RBG camera pixel coordinates to world coordinates, and then transform to pixel coordinates in the IR camera through world coordinates.

[0073] The specific steps are as follows:

[0074] I. Use Zhang Zhengyou calibration method to obtain the intrinsic parameters K of IR camera and RGB camera and the camera extrinsic parameter matrix M at a fixed distance, including rotation matrix R and translation vector T. Convert RGB pixel coordinate system to world coordinate system. The specific formula is as follows:

[0075]

[0076] [X W Y W Z W ] T : The real object position in the world coordinate system

[0077] M rgb : External parameter matrix of RGB camera

[0078] K rgb : The intrinsic parameter matrix of the RGB camera

[0079] S rgb : RGB camera scale factor

[0080] [u rgb v rgb ] T : Object position in RGB pixel coordinate system

[0081] II. The real coordinates of the object in the world coordinate system are obtained through formula (1), and then the real object coordinates in the world coordinate system are converted into pixel coordinates in the IR camera through the intrinsic parameter K and extrinsic parameter matrix M of the IR camera. The specific formula of the conversion is as follows:

[0082]

[0083] [X W Y W Z W ] T : The real object position in the world coordinate system

[0084] K ir : The intrinsic parameter matrix of the IR camera

[0085] M ir : External parameter matrix of IR camera

[0086] S ir : Scale factor of IR camera

[0087] [u ir v ir ] T : Object position in IR pixel coordinate system.

[0088] Step ③, data collection S3:

[0089] like Figure 2, install the built device on the pig restraining fence, and when the pig walks into the restraining fence, identify the pig's face. When the pig's face is in good condition, take a picture of the pig's face, including infrared thermal image, temperature matrix and RGB image. While taking the picture, send instructions to the environmental information sensor to obtain the current temperature, humidity, light intensity and wind speed information of the picture, and record them together.

[0090] Step ④, key point detection S4:

[0091] The improved YOLO target detection algorithm is used to detect the eyes, ear roots, forehead and nose of the pigs in the RGB image. The temperature information of the key points is obtained through the mapping relationship. The specific method is as follows: the captured image data is processed, the RGB image is detected by the target detection algorithm, and the key point area where the temperature needs to be obtained is located. The temperature information data in the infrared thermal image is obtained by the mapping method obtained by image registration. In order to improve the accuracy of the temperature in the subsequent model training, the average and maximum values ​​of the temperatures in the key point areas are calculated, and the error between the maximum value and the average value does not exceed 1 degree Celsius. If it exceeds, the current maximum value is removed and the local maximum value is searched again. In addition, since the temperature information obtained by the infrared thermal imaging device is not accurate enough, in order to comprehensively evaluate the temperature information, the temperature Euclidean distance of each key position is defined, and the calculation formula is as follows:

[0092]

[0093] T(a, b): The temperature difference between the two key areas a and b

[0094] a ave : Average temperature of area a

[0095] b ave : The average temperature of area b.

[0096] Step ⑤, model training S5:

[0097] For the training of the pig temperature detection model, a lightweight multi-layer perceptron neural network architecture is used. During training, the stochastic gradient descent (SGD) method is used as an optimizer to optimize the neural network. The input parameters include: the maximum and average temperatures at the pig's eyes, the maximum and average temperatures at the base of the ears, the maximum and average temperatures at the forehead, the maximum and average temperatures at the nose, as well as the Euclidean distance of the temperature at each position and environmental information. The model structure is as follows: Figure 3 shown.

[0098] Deploy the trained pig temperature detection model to edge computing devices so that the model can be applied in low-performance devices.

[0099] Step ⑥, model deployment S6

[0100] To implement real-time reasoning using the Jetson Orin NX development board, follow these steps:

[0101] ⅰ. Load the dual-camera control system and environmental information acquisition instructions into the development board;

[0102] ⅱ. The camera detects in real time whether there are live pigs in the video stream. When live pigs appear, it starts to determine whether the facial image of the live pig is clear. If it is clear, it captures the image data and environmental information data of this frame, analyzes the temperature information in the image, and transmits all parameters to the model reasoning function area;

[0103] ⅲ. Use the given parameters and the trained temperature detection model to infer the real body temperature information of the pig and display it on the page.

[0104] The specific operation process is as follows Figure 4 As shown:

[0105] a. Start - 401;

[0106] b. Video detection-402;

[0107] c. Determine whether there is live pig-403 in the picture, if yes, go to step d, otherwise jump to step b;

[0108] d. Get a frame - 404;

[0109] e. Determine whether the pig's face is clear-405, if yes, go to step f, otherwise jump to step b;

[0110] f. Obtain temperature information of key areas of live pigs - 406, and notify the system to obtain current environmental information - 400 and then enter step g;

[0111] g. Input the parameters into the pig temperature detection model-407;

[0112] h. Get pig temperature information - 408;

[0113] i. End of process - 409.

[0114] Step ⑦, body temperature detection S7

[0115] Install the device at the front end of the limiting fence to detect the body temperature of live pigs.

[0116] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can still modify the technical solutions described in the above embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Rapid detection method of pig body temperature based on image registration and multivariate information fusion, Features The steps include: ① Device construction (S1): construct a pig image information acquisition system and an environmental information acquisition system, wherein the image information acquisition system includes an IR camera and an RGB camera, and the environmental information acquisition system includes an environmental temperature and humidity sensor, a wind speed sensor, and a light intensity sensor; ② Image registration (S2): fix the relative position and shooting distance of the IR camera and the RGB camera, use the algorithm to calculate the mapping relationship between the two images, and complete the image registration; ③ Data collection (S3): The device is installed at the front end of the limiting fence to collect infrared thermal images, RGB images and environmental information of pig faces; ④ Key point detection (S4): Use the improved YOLO target detection algorithm to detect the eyes, ear roots, forehead and nose positions of the pig in the RGB image, and obtain the temperature information of the key points through the mapping relationship; ⑤ Model training (S5): The key point temperature information and environmental information obtained are made into a data set for deep learning model training to obtain a real-time pig body temperature detection model; ⑥Model deployment (S6): Deploy the model to the Jetson Orin NX development board to achieve real-time inference of the model on the embedded device; ⑦Body temperature detection (S7): The device is installed at the front end of the limiting fence to detect the body temperature of the pigs.

2. The method according to claim 1, Features The device in step ①: Including the working object - pig samples (7); The system is provided with a display (1), a Jetson Orin NX development board (2), an infrared and RGB camera (3), a wind speed sensor (4), a light intensity sensor (5), a limit fence (6), and a temperature and humidity sensor (8); Its location and connection relationship are: The live pig sample (7) is placed in a restraining fence (6), which is equipped with an infrared and RGB camera (3), a wind speed sensor (4), a light intensity sensor (5), and a temperature and humidity sensor (8). The Jetson Orin NX development board (2) is connected to the infrared and RGB camera (3), the wind speed sensor (4), the light intensity sensor (5), and the temperature and humidity sensor (8), and is also connected to an external display (1).

3. The method according to claim 1, Features The specific method for registering the infrared thermal image with the RGB image in step ② is: Ⅰ. Use Zhang Zhengyou calibration method to obtain the intrinsic parameters K of IR camera and RGB camera and the camera extrinsic parameter matrix M at a fixed distance, including rotation matrix R and translation vector T, to convert RGB pixel coordinate system into world coordinate system. The specific formula is as follows: [X W Y W Z W ] T : The real object position in the world coordinate system M rgb : External parameter matrix of RGB camera K rgb : The intrinsic parameter matrix of the RGB camera S rgb : RGB camera scale factor [u rgb v rgb ] T : Object position in RGB pixel coordinate system II. The real coordinates of the object in the world coordinate system are obtained through formula (1), and then the real object coordinates in the world coordinate system are converted into pixel coordinates under the IR camera through the intrinsic parameter K and extrinsic parameter matrix M of the IR camera. The specific formula for the conversion is as follows: [X W Y W Z W ] T : The real object position in the world coordinate system K ir : The intrinsic parameter matrix of the IR camera M ir : External parameter matrix of IR camera S ir : Scale factor of IR camera [u ir v ir ] T : Object position in IR pixel coordinate system.

4. The method according to claim 1, Features The specific method for obtaining the temperature information difference of the key points in step ④ is: define the temperature Euclidean distance of each key position, and the calculation formula is as follows: T(a, b): The temperature difference between the two key areas a and b a ave : Average temperature of area a b ave : The average temperature of area b.

5. The method according to claim 1, Features The specific operation process of step ⑥ is as follows: a. Start (401); b. Video detection (402); c. Determine whether there are live pigs in the image (403). If yes, go to step d; otherwise, jump to step b. d. Get a frame (404); e. Determine whether the pig's face is clear (405), if so, proceed to step f, otherwise jump to step b; f. Obtain temperature information of key areas of live pigs (406), and notify the system to obtain current environmental information (400) and enter step g; g. Input the parameters into the pig temperature detection model (407); h. Obtaining pig temperature information (408); i. The process ends (409).

Images