A silage harvester living body detection and early warning method fusing thermal infrared and visible light

Abstract

The present application relates to the field of agricultural machinery safety perception and intelligent early warning technology, and particularly relates to a silage harvester living body detection and early warning method fusing thermal infrared and visible light, wherein a thermal infrared camera and a visible light camera are installed on the roof of the harvester, calibration parameters and spatial mapping relationships are obtained in advance, dual-mode images of the front work area are collected in real time, pre-stored parameters are called to perform synchronous matching and spatial registration, the dual-mode image pair is input into an RGBT-YOLO network based on the improvement of YOLOv11, temperature features are extracted through the thermal infrared branch, texture features are extracted through the visible light branch, and the class, position and confidence of the living body target are output after cross-modal fusion and lightweight detection, when a person or an animal enters a preset dangerous area in front of the header and continuously meets the alarm conditions, an audible and visual alarm device is driven to send an early warning signal, and the problems of insufficient consideration of dual-mode image registration, feature-level fusion detection and early warning control closed loop in the working scene of the dangerous area in front of the silage harvester at the present stage are solved.

Description

Technical Field

[0001] This invention relates to the field of agricultural machinery safety sensing and intelligent early warning technology, specifically to a live detection and early warning method for silage harvesters that integrates thermal infrared and visible light. Background Technology

[0002] Silage harvesters are essential equipment for the mechanized harvesting of silage corn, capable of continuously performing operations such as crop cutting, feeding, chopping, and scattering. Due to the large size of self-propelled silage harvesters, with the header and feeding device located at the front, the operator relies primarily on the cab's visibility to observe the area ahead. However, mature silage corn plants are tall with broad leaves and a closed canopy in the field, making it difficult for operators to spot people or animals entering the work area, thus posing a significant safety risk.

[0003] Existing agricultural machinery safety inspection methods mostly employ single visible light cameras, single thermal infrared cameras, ultrasonic sensors, or radar. Single visible light cameras can acquire rich color, texture, and morphological information, but are easily affected by changes in lighting, shadows, dust, and plant shading. Single thermal infrared cameras can reflect the temperature characteristics of living targets such as people or animals, but the image texture and edge details are insufficient, and false detections are prone to occur under high-temperature backgrounds or heat source interference conditions. Other non-contact sensors can achieve obstacle detection to a certain extent, but they have problems such as high cost, insufficient target category recognition ability, or limited adaptability to crop shading scenarios.

[0004] Existing technologies include methods that combine thermal infrared images with visible light images for field life target detection. However, these methods often focus on image input, time synchronization, image stabilization, or lightweight processing of conventional target detection models, and do not adequately consider dual-modal image registration, feature-level fusion detection, and early warning control closed-loop in dangerous operating scenarios in front of silage harvesters.

[0005] In summary, this invention proposes a live animal detection and early warning method for silage harvesters that integrates thermal infrared and visible light to solve the problems mentioned in the background art. Summary of the Invention

[0006] The purpose of this invention is to provide a live detection and early warning method for silage harvesters that integrates thermal infrared and visible light, making full use of the temperature information of thermal infrared images and the texture and morphology information of visible light images to achieve real-time detection and early warning of human and animal targets in complex field environments.

[0007] A live detection and early warning method for silage harvesters that integrates thermal infrared and visible light, the specific steps of which are as follows:

[0008] Step S1: Install a thermal infrared camera and a visible light camera on the roof of the silage harvester to collect images of the working area in front of the silage harvester;

[0009] Step S2: Pre-acquire the calibration parameters of the thermal infrared camera and the visible light camera, as well as the spatial mapping relationship between the two cameras, and save the calibration parameters and spatial mapping relationship to the host computer;

[0010] Step S3: Collect thermal infrared and visible light images of the working area in front of the silage harvester, respectively;

[0011] Step S4: Call the pre-saved calibration parameters and spatial mapping relationship to perform synchronous matching and spatial registration of the thermal infrared image and the visible light image to obtain a dual-modal image pair;

[0012] Step S5: Input the dual-modal image pair into the RGBT-YOLO live target detection network based on YOLOv11 improvement, extract and fuse the temperature features in the thermal infrared image and the texture morphology features in the visible light image, and output the category, location and confidence of the live target;

[0013] Step S6: Determine whether the alarm conditions are met based on the category, location, and confidence level of the live target.

[0014] Step S7: When the number of frames in which the live target continuously meets the alarm conditions reaches the preset alarm frame number, an alarm control command is sent to the main control unit, and the audible and visual alarm device is driven to issue a warning signal.

[0015] Step S8: When the number of consecutive frames in which a live target fails to meet the alarm conditions reaches the preset number of frames to be deactivated, the alarm state is deactivated.

[0016] Further specifying, the calibration parameters in step S2 include thermal infrared camera intrinsic parameters, visible light camera intrinsic parameters, thermal infrared camera distortion parameters, and visible light camera distortion parameters. The spatial mapping relationship is obtained based on the relative pose relationship between the thermal infrared camera and the visible light camera. The spatial mapping relationship is that the spatial points are first transformed from the world coordinate system to the camera coordinate system through the extrinsic parameter relationship, and then perspective projection is completed according to the camera intrinsic parameters, and finally mapped to the image pixel coordinate system.

[0017] Further defined, the RGBT-YOLO live target detection network in step S5 includes a thermal infrared image input branch, a visible light image input branch, a dual-modal feature extraction unit, a channel-level cross-modal feature fusion Transformer module, an adaptive feature attention module, a depth convolution module, and a detection output unit. The thermal infrared image input branch is used to extract the temperature difference features of the live target, and the visible light image input branch is used to extract the texture, edge, and morphological features of the live target.

[0018] Further defined, the channel-level cross-modal feature fusion Transformer module is positioned between the intermediate feature layers of thermal infrared and visible light features, and is used to perform channel-level interaction and fusion of thermal infrared and visible light features, so that the temperature response information in the thermal infrared image and the texture morphology information in the visible light image complement each other. The adaptive feature attention module is used to adaptively allocate weights according to the contribution of features at different scales to detection, enhance the features of the live target region and suppress background interference. The deep convolution module is used to replace some of the standard convolutions in the network to reduce the number of parameters and computation.

[0019] Further specifying, in step S6, when the live target category is a person or animal, the detection confidence level is greater than the preset confidence threshold, and the center point of the detection frame or the main area of ​​the detection frame is located in the preset danger zone in front of the silage harvester, the live target is determined to meet the alarm conditions; the preset danger zone is the area within 3 meters in front of the cutting platform of the silage harvester, and this distance threshold is determined comprehensively based on the operating speed and emergency response time, and can be adaptively adjusted according to the real-time driving speed.

[0020] Furthermore, when multiple live targets are simultaneously located within a preset danger zone, an alarm control command will be triggered if any live target meets the alarm conditions, without distinguishing between the priorities of humans and animals.

[0021] Further defined, in step S7, when a live target meets the alarm conditions within a consecutive preset alarm frame number, the live target is determined to be a valid warning target and an alarm is triggered; in step S8, when a live target does not meet the alarm conditions within a consecutive preset de-alarm frame number, the alarm state is deactivated; the preset confidence threshold, the consecutive preset alarm frame number, and the consecutive preset de-alarm frame number are all settable parameters.

[0022] Further specified, the preset confidence threshold is 80%, the number of consecutive preset alarm frames is 2 frames, and the number of consecutive preset deactivation frames is 3 frames.

[0023] Further specifying, in step S7, the host computer generates an alarm control command based on the alarm judgment result and sends the alarm control command to the main control unit. The main control unit controls the relay module to turn on, so that the audible and visual alarm device emits a warning signal. In step S8, when the alarm state is cleared, the host computer sends an alarm clearing command to the main control unit. The main control unit controls the relay module to turn off, so that the audible and visual alarm device stops working.

[0024] Furthermore, the main control unit can also be connected to the vehicle braking interface or the power cut-off interface of the working component. When the live target is continuously in the preset danger zone, the main control unit outputs a control signal to the vehicle braking interface or the power cut-off interface of the working component to assist the silage harvester in decelerating, stopping, or cutting off the power of the working component.

[0025] The advantages of this invention compared to the prior art are as follows:

[0026] 1. By simultaneously utilizing the temperature information from thermal infrared images and the texture and morphology information from visible light images, the stability of silage harvesters in detecting human and animal targets can be improved under conditions of plant shading, light changes, high temperatures, and nighttime.

[0027] 2. The calibration and registration of the thermal infrared camera and the visible light camera are included in the system initialization. During real-time detection, only the pre-saved calibration parameters and spatial mapping relationships are called, which avoids repeated calibration during the operation and improves the engineering rationality and real-time performance of the method.

[0028] 3. An improved RGBT-YOLO live target detection network based on YOLOv11 is adopted. Temperature features and texture features are extracted through thermal infrared image input branch and visible light image input branch, respectively. Cross-modal feature fusion, target region enhancement and lightweight detection are realized through CCFT module, AFAM module and DWConv module, which improves the live target detection capability in complex field environment.

[0029] 4. By making early warning judgments based on target category, location, confidence level, and continuous frame detection results, false alarms caused by single-frame false detections can be reduced, and the stability of the system's early warning can be improved.

[0030] 5. The preset confidence threshold, the number of consecutive preset alarm frames, and the number of consecutive preset cancellation frames are all settable parameters. They can be adjusted according to the operation speed, the complexity of the field environment, the stability of the model output, and the sensitivity of the early warning, thus having good adaptability. Attached Figure Description

[0031] Figure 1 This is a flowchart of the liveness detection and early warning method of the present invention;

[0032] Figure 2 This is a flowchart of the early warning control system of the present invention;

[0033] Figure 3 This diagram illustrates the transformation relationships between the world coordinate system, camera coordinate system, and pixel coordinate system. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] Example:

[0036] like Figures 1-3 As shown, a live detection and early warning method for silage harvesters that integrates thermal infrared and visible light is described, with the following specific steps:

[0037] Step S1: After the installation and testing of the silage harvester are completed, install a thermal infrared camera and a visible light camera on the roof of the silage harvester to collect images of the working area in front of the harvester.

[0038] Step S2: After installation, calibrate the thermal infrared camera and the visible light camera to obtain the intrinsic parameters of the thermal infrared camera, the intrinsic parameters of the visible light camera, the distortion parameters of the thermal infrared camera, and the distortion parameters of the visible light camera. Simultaneously, based on the relative position and orientation of the thermal infrared camera and the visible light camera, obtain the spatial mapping relationship between the thermal infrared image and the visible light image. This spatial mapping relationship is used to establish the relationship between corresponding regions or corresponding pixels in the two types of images. The spatial mapping relationship involves first transforming a spatial point from the world coordinate system to the camera coordinate system through extrinsic parameter relationships, then performing perspective projection based on the camera's intrinsic parameters, and finally mapping it to the image pixel coordinate system. Let the coordinates of the spatial point P in the world coordinate system be (X...). w , Y w Z w Then, the coordinates of this point in the camera coordinate system after extrinsic parameter transformation can be expressed as:

[0039]

[0040] Where R is the rotation matrix and t is the translation vector, representing the camera's pose and position relative to the world coordinate system, respectively;

[0041] By further combining perspective projection relationships, a complete mapping model of spatial points from the world coordinate system to the pixel coordinate system can be obtained:

[0042]

[0043] Where λ is the scale factor, which is proportional to the depth of the spatial point in the camera coordinate system, and K is the camera intrinsic parameter matrix, which includes parameters such as focal length and principal point position. The core task of camera calibration is to solve for the intrinsic parameter K and the extrinsic parameter (R, t) to establish the geometric mapping relationship between the spatial point and the image pixels;

[0044] The calibration parameters and spatial mapping relationship are acquired during the system initialization phase and saved to the host computer. When the silage harvester is actually operating, it no longer performs camera calibration repeatedly, but directly calls the pre-saved calibration parameters and spatial mapping relationship for image correction and registration.

[0045] Step S3: After the silage harvester enters the working state, thermal infrared images and visible light images of the working area in front of the silage harvester are collected respectively. The images collected by the thermal infrared camera are used to reflect the temperature difference between the personnel or animal target and the surrounding crops and soil background; the images collected by the visible light camera are used to reflect the texture, edge and morphological information of the personnel or animal target.

[0046] Step S4: After receiving the two images, the host computer performs synchronous matching of the thermal infrared image and the visible light image according to the frame sequence number, acquisition time or video alignment result. It calls the pre-saved calibration parameters and spatial mapping relationship to combine images corresponding to the same operation time or adjacent operation time. Then, it calls the calibration parameters and spatial mapping relationship saved in the system initialization stage to perform distortion correction, size matching and spatial registration on the thermal infrared image and the visible light image to obtain a dual-modal image pair.

[0047] Step S5: Input the dual-modal image pair into the RGBT-YOLO live target detection network based on YOLOv11 (this network adopts a dual-input structure, one input is a thermal infrared image, and the other input is a visible light image). After the thermal infrared image and the visible light image are processed by the corresponding feature extraction branches, thermal infrared features and visible light features at different scales are obtained. The temperature features in the thermal infrared image and the texture morphology features in the visible light image are extracted and fused to output the category, location and confidence of the live target.

[0048] The thermal infrared image input branch focuses on extracting temperature difference features between personnel or animal targets and the background of crops, soil, and machinery, while the visible light image input branch focuses on extracting texture, edge, contour, and morphological information of personnel or animal targets. Due to the presence of plant shading, cluttered backgrounds, and changes in lighting in silage corn fields, single modality is prone to false detection or missed detection. Therefore, this embodiment enhances the ability to express live targets through dual-modality feature fusion.

[0049] In the intermediate feature layer, thermal infrared features and visible light features are input into the channel-wise Cross Fusion Transformer (CCFT) module. The CCFT module performs channel-level interaction and fusion of the two types of features, so that the temperature response information provided by the thermal infrared mode and the texture and morphology information provided by the visible light mode complement each other. For areas in the thermal infrared image with obvious response but insufficient texture, edge and morphology information can be supplemented by visible light features; for areas in the visible light image that are obviously affected by light or plant interference, temperature response information can be supplemented by thermal infrared features.

[0050] The fused features are input into an Adaptive Feature Attention Module (AFAM). The AFAM module assigns weights based on the contribution of fused features at different scales to the live target detection task, improving the feature response of the area where people or animals are located and reducing the influence of background areas such as crop leaves, straw, soil, and mechanical structures. Simultaneously, a Deepwise Convolution (DWConv) module is introduced into the RGBT-YOLO live target detection network. The DWConv module replaces some standard convolutional structures in the network, reducing the number of model parameters and computational cost, and improving real-time detection speed. Finally, the detection output unit outputs the category of the person or animal target, the bounding box position, and the confidence score based on the multi-scale fused features. The bounding box position includes the bounding box coordinates or the coordinates of the bounding box center point, and the confidence score characterizes the reliability of the detection result and serves as the basis for subsequent alarm condition judgment.

[0051] Step S6: Based on the category, location, and confidence level of the live target, determine whether it meets the alarm conditions. After receiving the target category, detection box position, and confidence level output by the RGBT-YOLO live target detection network, the host computer first determines whether a human or animal target is detected. If no human or animal target is detected, continue to acquire the next frame image. If a human or animal target is detected, further determine whether the target confidence level reaches the preset confidence threshold. The preset confidence threshold is a settable parameter that can be adjusted according to the operating speed of the silage harvester, the complexity of the field environment, the stability of the detection model output, and the warning sensitivity. When the target confidence level meets the requirements, further determine whether the target is located in the preset danger zone in front of the silage harvester. The preset danger zone can be set according to the operating range in front of the silage harvester, the width of the header, the operating speed, and the safety distance. When the target category, confidence level, and danger zone position all meet the conditions, determine that the current frame meets the alarm conditions, and count the consecutive frames that meet the alarm conditions.

[0052] Step S7: When the number of frames in which a live target continuously meets the alarm conditions reaches the preset alarm frame number, an alarm control command is sent to the main control unit, and the audible and visual alarm device is driven to issue a warning signal. The target is determined to be a valid warning target, and the system enters the alarm execution process. The preset confidence threshold is 80%, the number of consecutive preset alarm frames is 2 frames, and the number of consecutive preset deactivation frames is 3 frames. When the conditions are met, the alarm is triggered.

[0053] Step S8: When the number of consecutive frames in which a live target fails to meet the alarm conditions reaches the preset number of frames to be deactivated, the alarm state is deactivated.

[0054] When the system determines that a valid warning target exists, the host computer sends an alarm control command to the main control unit. After receiving the alarm control command, the main control unit controls the relay module to turn on, activating the audible and visual alarm device and emitting an audible and visual alarm signal. The system then enters alarm mode and continues to monitor the working area in front of the silage harvester.

[0055] If human or animal targets are still detected and the target confidence level reaches the preset confidence threshold, the system will remain in alarm mode; if no human or animal targets meeting the alarm conditions are detected for several consecutive frames, or if the target leaves the preset danger zone, the danger state will be determined to be lifted.

[0056] When the number of consecutive frames that do not meet the alarm conditions reaches the preset number of frames to be deactivated, the host computer sends an alarm deactivation command to the main control unit. The main control unit then controls the relay module to disconnect, causing the audible and visual alarm device to stop working. The alarm is deactivated when the alarm conditions are not met for three consecutive frames.

[0057] The preset confidence threshold, the number of consecutive preset alarm frames, and the number of consecutive preset cancellation frames are all settable parameters. These parameters can be set to other values ​​based on the actual operating speed, target detection stability, and early warning sensitivity.

[0058] Example 2:

[0059] In Example 2, the main control unit can also be connected to the vehicle braking interface or the power cut-off interface of the working components. When personnel or animals remain in the preset danger zone, the main control unit outputs a control signal to assist the silage harvester in decelerating, stopping, or cutting off the power of the cutting platform, feeding device, and other working components. If no effective warning target is detected, or the detected target has not entered the preset danger zone, the system continues to acquire the next frame of thermal infrared image and visible light image, and repeats the synchronous matching, spatial registration, fusion detection, alarm condition judgment, and warning control process.

[0060] The above provides a detailed description of a live animal detection and early warning method for silage harvesters that integrates thermal infrared and visible light. The specific embodiments are only used to help understand the method and core ideas of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for live detection and early warning of silage harvesters that integrates thermal infrared and visible light, characterized in that: The specific steps are as follows: Step S1: Install a thermal infrared camera and a visible light camera on the roof of the silage harvester to collect images of the working area in front of the silage harvester; Step S2: Pre-acquire the calibration parameters of the thermal infrared camera and the visible light camera, as well as the spatial mapping relationship between the two cameras, and save the calibration parameters and spatial mapping relationship to the host computer; Step S3: Collect thermal infrared and visible light images of the working area in front of the silage harvester, respectively; Step S4: Call the pre-saved calibration parameters and spatial mapping relationship to perform synchronous matching and spatial registration of the thermal infrared image and the visible light image to obtain a dual-modal image pair; Step S5: Input the dual-modal image pair into the RGBT-YOLO live target detection network based on YOLOv11 improvement, extract and fuse the temperature features in the thermal infrared image and the texture morphology features in the visible light image, and output the category, location and confidence of the live target; Step S6: Determine whether the alarm conditions are met based on the category, location, and confidence level of the live target. Step S7: When the number of frames in which the live target continuously meets the alarm conditions reaches the preset alarm frame number, an alarm control command is sent to the main control unit, and the audible and visual alarm device is driven to issue a warning signal. Step S8: When the number of consecutive frames in which a live target fails to meet the alarm conditions reaches the preset number of frames to be deactivated, the alarm state is deactivated.

2. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 1, characterized in that: The calibration parameters in step S2 include the intrinsic parameters of the thermal infrared camera, the intrinsic parameters of the visible light camera, the distortion parameters of the thermal infrared camera, and the distortion parameters of the visible light camera. The spatial mapping relationship is obtained based on the relative pose relationship between the thermal infrared camera and the visible light camera. The spatial mapping relationship is that the spatial points are first transformed from the world coordinate system to the camera coordinate system through the extrinsic parameter relationship, and then perspective projection is completed according to the camera intrinsic parameters, and finally mapped to the image pixel coordinate system.

3. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 1, characterized in that... The RGBT-YOLO live target detection network in step S5 includes a thermal infrared image input branch, a visible light image input branch, a dual-modal feature extraction unit, a channel-level cross-modal feature fusion Transformer module, an adaptive feature attention module, a depth convolution module, and a detection output unit. The thermal infrared image input branch is used to extract the temperature difference features of the live target, and the visible light image input branch is used to extract the texture, edge, and morphological features of the live target.

4. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 3, characterized in that: The channel-level cross-modal feature fusion Transformer module is positioned between the intermediate feature layers of thermal infrared and visible light features. It is used to perform channel-level interaction and fusion of thermal infrared and visible light features, making the temperature response information in the thermal infrared image complementary to the texture and morphology information in the visible light image. The adaptive feature attention module is used to adaptively allocate weights according to the contribution of features at different scales to detection, enhancing the features of the live target region and suppressing background interference. The deep convolution module is used to replace some of the standard convolutions in the network to reduce the number of parameters and computational cost.

5. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 1, characterized in that: In step S6, when the live target is classified as a person or animal, the detection confidence level is greater than the preset confidence threshold, and the center point of the detection frame or the main area of ​​the detection frame is located in the preset danger zone in front of the silage harvester, the live target is determined to meet the alarm conditions. The preset danger zone is the area within 3 meters in front of the cutting platform of the silage harvester. This distance threshold is determined based on the operating speed and emergency response time, and can be adaptively adjusted according to the real-time driving speed.

6. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 5, characterized in that: When multiple live targets are simultaneously located within a preset danger zone, an alarm control command will be triggered if any live target meets the alarm conditions, regardless of the priority of personnel or animals.

7. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 1, characterized in that: In step S7, when a live target meets the alarm conditions within a consecutive preset alarm frame number, the live target is determined to be a valid warning target and an alarm is triggered. In step S8, when a live target does not meet the alarm conditions within a consecutive preset de-alarm frame number, the alarm state is deactivated. The preset confidence threshold, the consecutive preset alarm frame number, and the consecutive preset de-alarm frame number are all settable parameters.

8. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 7, characterized in that: The preset confidence threshold is 80%, the preset number of consecutive alarm frames is 2 frames, and the preset number of consecutive deactivation frames is 3 frames.

9. The method for live detection and early warning of silage harvesters integrating thermal infrared and visible light as described in claim 1, characterized in that: In step S7, the host computer generates an alarm control command based on the alarm judgment result and sends the alarm control command to the main control unit. The main control unit controls the relay module to turn on, so that the audible and visual alarm device issues a warning signal. In step S8, when the alarm state is cleared, the host computer sends an alarm clearing command to the main control unit. The main control unit controls the relay module to turn off, so that the audible and visual alarm device stops working.

10. The method for live animal detection and early warning of a silage harvester integrating thermal infrared and visible light as described in claim 5, characterized in that: The main control unit can also be connected to the vehicle braking interface or the power cut-off interface of the working component. When the live target is continuously in the preset danger zone, the main control unit outputs a control signal to the vehicle braking interface or the power cut-off interface of the working component to assist the silage harvester in decelerating, stopping or cutting off the power of the working component.

Images