Target recognition system based on multi-modal image analysis

Abstract

The application discloses a target recognition system based on multi-modal image analysis, relates to the technical field of intelligent sensing and behavior recognition in agriculture, and is used for solving the problems of behavior misjudgment and insufficient recognition stability of agricultural behaviors in a facility environment under the influence of mirror interference and non-synchronous multi-modal data. The system comprises a multi-modal data acquisition module, a data preprocessing and fusion module, a sample library and index system module, and a visualized tracing and decision support module. The system is characterized in that a time window is set for frame-by-frame scanning in a strong interference reflection scene, a mirror mask is constructed, and the main body is marked based on consistency scoring; image and sensing data are fused to establish synchronous binding; a double-channel SlowFast network and an SMC-ECA attention mechanism are used to extract behavior characteristics; a three-layer index system and a behavior database are constructed to realize full-process behavior tracing and intelligent decision assistance.

Description

Technical Field

[0001] This invention relates to the field of agricultural intelligent sensing and behavior recognition technology, and more specifically, to a target recognition system based on multimodal image analysis. Background Technology

[0002] Against the backdrop of rapid development in digital agriculture, automated agricultural behavior recognition has become a crucial foundation for improving operational standardization and efficiency. However, existing technologies largely rely on single image channels and static perspectives, making it difficult to address mirror interference caused by high-reflectivity surfaces in facility agriculture. This is particularly true for fine-grained behaviors such as fruit thinning and spraying, which require accurate identification of contact, traction, and separation processes, easily leading to ontology / mirror image confusion, incorrect action segmentation, or duplicate counting. Furthermore, the coupling relationship between environmental parameters and behaviors is not adequately modeled; for example, real-time linkage analysis methods are lacking for fertilization and conductivity transitions, and spraying and CO2 mutations. Simultaneously, existing methods are weak in synchronously binding multimodal data, indexing behavioral segments, and tracing operational paths across different sites, lacking decision feedback channels for anomaly handling and strategy recommendations.

[0003] To address the above problems, this invention proposes a solution. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a target recognition system based on multimodal image analysis to address the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In a preferred embodiment, it includes: a multimodal data acquisition module, a data preprocessing and fusion module, a sample library and indicator system module, and a visualization traceability and decision support module, with signal connections between the modules;

[0007] The multimodal data acquisition module is used to slowly scan along the normal side of the reflection plane by a slide rail or pan-tilt unit at a low angle, label continuous frame segments frame by frame and calculate the time length baseline. Based on the time length baseline, the closing time length of the greenhouse reflection composite interference is determined, and the sliding time window is set with the closing time length of the greenhouse reflection composite interference. The reflection plane parameters and mirror candidate masks are constructed by the reflection plane equation, normal vector and reflection homography mapping matrix, and the body and mirror are classified and labeled according to the consistency score.

[0008] The data preprocessing and fusion module is used to construct a multi-source synchronous binding relationship based on the main image frame through image filtering and enhancement, sensor signal reconstruction and formatting, and output a fusion vector;

[0009] The sample library and indicator system module is used to extract temporal features by inputting the fusion vector into the dual-channel SlowFast network and performing modality reweighting, and then combine it with GRU to output behavior classification labels;

[0010] The visualization traceability and decision support module is used to generate behavior trajectory layers and summary tables by building a data pipeline that runs through the identification results and indicator evaluation, and output scheduling suggestions and disposal plans based on the inference engine.

[0011] In a preferred embodiment, a variable focal length high-resolution CMOS image sensing module is deployed and the elevation angle is calibrated with a ground reference point and a sensing cone. Then, a multi-band near-infrared imaging unit is installed and frame-level matching is performed.

[0012] Then, the variable focal length high-resolution CMOS image sensing module and the multi-band near-infrared imaging unit are connected to the edge image buffer array. After high-pass filtering and timestamp recalibration, a frame-level binding relationship is established between the main image frame sequence and the frame-level matching result of the multi-band near-infrared imaging unit. A gray-scale channel image is constructed based on the multi-band near-infrared imaging unit, and an exposure equalization reference window is set in the main stem area of ​​the crop to determine the action criteria.

[0013] In a preferred embodiment, geometric and optical joint calibration is performed during the deployment phase. After all optoelectronic devices are connected to the edge image buffer array, high-pass band filtering and timestamp recalibration are performed. The slide rail or gimbal slowly scans along the normal side of the reflection plane at a low angle. Frame by frame, continuous frame segments appearing due to the superposition of the reflection surface and low-angle incident light, the staggered framing of the main view and the opposite side view, and the lag caused by exposure adaptation and supplementary light feedback are marked.

[0014] Then, the corresponding feature durations are calculated based on the marked start and end frames, and the largest of the feature durations is taken as the time length baseline. When the virtual and real symmetrical overlaps do not fall back or the hysteresis overlaps are not cleared at the end of the time length baseline, the frame is extrapolated once according to the slow scan and the pause frame index until the frame that simultaneously satisfies the disappearance of virtual and real symmetrical overlaps and the clearing of hysteresis overlaps is reached for the first time. The extrapolated time length baseline is then determined as the closing time length of the shed reflection composite interference.

[0015] In a preferred embodiment, by setting the sliding time window N1 with the closing time length of the shed reflection composite interference, the short-wave infrared high-light ridge is scanned in small steps along the normal side of the reflection plane to construct the reflection plane equation, normal vector and reflection homography mapping matrix, and establish a snapshot of the on-duty parameters including the reflection plane equation, normal vector, reflection homography mapping matrix, incident angle sequence, supplementary lighting state and camera attitude.

[0016] In a preferred embodiment, during the acquisition and alignment stage, the main camera module, the opposite fixed-position camera, and the multi-band near-infrared imaging unit are uniformly connected to the edge image buffer array; wherein, the main camera module forms the main view, the opposite fixed-position camera forms the opposite side view, and the multi-band near-infrared imaging unit is used to construct a short-wave infrared grayscale channel image.

[0017] The frame number alignment and timestamp recalibration are completed through synchronous triggering, and the polarization difference or normalized specular indicator is calculated in the short-wave infrared grayscale channel to generate a candidate mirror mask.

[0018] In a preferred embodiment, normalized cross-correlation and spatial position consistency judgment are performed on the main view candidate and the projection point of the mirror domain to form a consistency score. If the spatial judgment is true and the target area is not covered by specular highlights, it is marked as the main body candidate. Otherwise, it is stripped as the mirror and the exposure and gain time axis of this stage is reconstructed after the reflection decoupling channel is triggered.

[0019] In a preferred embodiment, environmental sensing terminals with multimodal sampling capabilities are symmetrically installed in the visible area of ​​the camera equipment to collect signals of soil moisture, electrical conductivity, pH, spraying flow rate and meteorological factors, and are uniformly connected to the main control ADC module for synchronous sampling through an analog signal conditioning channel.

[0020] Before sampling, a channel status scan is performed. During the initial deployment, response curves of various sensors are read and compensation parameters are established. These parameters are written into Flash to form a calibration benchmark and to construct a correlation description between operational behavior and environmental factors.

[0021] In a preferred embodiment, frame-by-frame bilateral filtering and Retinex enhancement are performed on the acquired image sequence, and image partitioning and parallel processing are performed in the edge node FPGA array; after sampling the outputs of various environmental and crop sensors by ADC, moving average, first-order difference and cubic spline interpolation reconstruction are performed in sequence to eliminate signal discontinuities and abrupt disturbances.

[0022] Construct a multi-source synchronization binding relationship, perform frame alignment between image frames and sensor group data, and insert zero-order hold or null label compensation for delay sources.

[0023] In a preferred embodiment, the fused vector is input into a dual-channel SlowFast feature extraction network. The Slow path captures the operation rhythm and keyframe semantic transfer, while the Fast path extracts high-frequency short-term dynamic features. Based on this, each modality dimension is dynamically reweighted. Subsequently, the deep feature sequence is smoothed using a GRU network, and pre- and post-warm-up zones and delay windows are set to prevent erroneous cutting.

[0024] In a preferred embodiment, a visual data pipeline is constructed that runs through identification output, indicator evaluation, and behavior backtracking. The main frame of the image, the sequence curve of the sensor value, the time history of the environmental indicator, and the indicator score are associated to generate a nested time-series behavior trajectory layer and a behavior evaluation summary table. An operation hotspot monitoring module and a behavior event heat map are constructed, and policy recommendations, personnel scheduling strategies, or behavior handling plans are output based on the platform's inference engine.

[0025] The technical effects and advantages of the target recognition system based on multimodal image analysis of this invention are as follows:

[0026] This invention achieves precise separation of the mirror image by setting the reflection interference closing time and annotating the highlight ridges, symmetrical overlap, and strong region drift frame by frame, significantly improving the recognition accuracy in complex reflection scenarios. It introduces frame-level binding and multi-source synchronization mechanisms to achieve consistent alignment between the image and sensor signals, effectively eliminating temporal drift. Combining SlowFast and SMC-ECA structures, it completes multimodal feature extraction, enhancing the model's ability to discriminate weak feature behaviors. By constructing a multi-dimensional indicator system and a visualized trajectory layer, it enables full-process monitoring, evaluation, and backtracking of the operation, and generates practical policy recommendations through an inference engine, addressing the shortcomings of traditional solutions in interference suppression, quality evaluation, and decision support. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the target recognition system module based on multimodal image analysis according to the present invention.

[0028] Figure 2 This is a timing diagram of the target recognition system based on multimodal image analysis according to the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example

[0031] This invention discloses a target recognition system based on multimodal image analysis, such as... Figure 1 As shown, it includes: a multimodal data acquisition module, a data preprocessing and fusion module, a sample library and indicator system module, and a visualization traceability and decision support module, with signal connections between each module.

[0032] First, such as Figure 2As shown, in order to achieve continuous identification and multi-source inversion of key operational behaviors in agricultural production, the front-end deployment prioritizes the deployment of high-resolution CMOS image sensing modules with variable focal length. These modules are fixedly installed on the horizontal support above the work area by adding a spherical protective cover and an anti-glare filter. The pitch angle is dynamically set by calibrating the ground reference point and sensing cone during the debugging phase to cover the complete human operation range and crop interaction area.

[0033] Automatic shutter compensation and frame rate stabilization modules are embedded in the image path of the camera module, and an exposure equalization reference window is set in the main stem area of ​​the crop to ensure that the outline of the person can still be captured with stable contrast even under fluctuating ambient light. For example, in continuous sowing operations, the system can identify the boundaries of the sowing rhythm by the frequency of hand occlusion and the rate of change in background texture contrast.

[0034] At the opposite edge of the work area, in conjunction with the height of the crop canopy, a low-angle incident multi-band near-infrared imaging unit is installed, with its optical axis forming a cross angle with the main camera module.

[0035] A fixed-position camera is set at a fixed position on the opposite side of the visible area of ​​the camera device. The fixed-position camera on the opposite side forms the opposite side view, which is used to interpolate the main view formed by the main camera module. The alignment between the main view and the opposite side view is determined by combining the reflection homography mapping.

[0036] This device constructs grayscale channel images based on changes in shortwave infrared reflectance intensity and performs frame-level matching with the main image acquisition via a synchronous trigger line. In a typical fruit thinning scenario, the reflectance of fruits and background leaves is distinguishable in the near-infrared band, and combined with the hand trajectory in the main view, it can form a highly reliable basis for action judgment. Then, all optoelectronic devices are uniformly connected to the edge image buffer array, low-frequency jitter is removed by a high-pass band filter module, and timestamp recalibration is completed in the reconstructed frame.

[0037] To achieve consistent binding between the main camera module and the multi-band near-infrared imaging unit on the time axis, let the main image frame sequence be {( , )}, multimodal sensing sampling is {( , If )}, then establish a frame-level binding relationship m*(k)= | - |;When| - |≤ Synchronization is considered successful if the condition is met, and the binding is written to the frame; otherwise, zero-order hold or a null label is inserted for s. Set by the current shift parameter snapshot.

[0038] In a facility with a highly reflective film or glass facade, when a sliding camera and a fixed camera on the opposite side frame the image, the mirror image generated by the reflective surface and the actual object exhibit highly coherent appearance and movement within a short time window. When the gimbal or sliding track slowly scans near the reflective surface and the supplementary lighting or incident angle changes, the mirror image trajectory overlaps with the actual object across the field of view. It should be noted that when performing low-angle incident imaging near the highly reflective film or glass facade of the facility, only in this scenario will the following three types of composite interference conditions occur simultaneously:

[0039] The short-wave infrared highlight ridge line near the normal direction of the reflecting plane drifts slowly with the incident angle;

[0040] After homography, the front view and the opposite side view appear to coincide in a symmetrical manner;

[0041] There is a slight lag in the exposure adaptation and fill light feedback, which overwrites the true contours.

[0042] Meanwhile, in image recording, this manifests as follows: the hand contour in the main grayscale channel forms a symmetrical double band with the highlight ridge; the projection of the fixed camera trajectory on the opposite side overlaps with the main camera trajectory on the reflective surface with hysteresis; and the reflectance of the fruit / leaf in the short-wave infrared channel within the short time window is reversed in the mirror domain, thus causing misreading of the contact, traction, and separation timing of fruit thinning actions.

[0043] Existing techniques use brightness threshold highlight suppression, morphological opening and closing, and cross-frame non-maximum suppression to remove duplicate targets, and use fixed homography or static extrinsic parameters for multi-camera alignment. However, they fail to incorporate geometric constraints on the reflection plane, model short-wave infrared reflection under varying incident angles, distinguish between specular and diffuse reflection using polarization splitting, or include exposure adaptation and supplementary lighting feedback lag in the linkage correction. This leads to mirrored hands and fruits being mistakenly stitched to the original hand and the original fruit main chain; actions being repeatedly counted; the fruit stalk breakage time being moved forward or backward; and the credibility of actions being abnormally inflated, contaminating rule thresholds and model weights.

[0044] Therefore, in this embodiment, geometric and optical joint calibration is performed first during the deployment phase. Specifically, frame-level matching is first completed by synchronizing the trigger line with the main image acquisition, all optoelectronic devices are uniformly connected to the edge image buffer array, high-pass band filtering is performed to remove low-frequency jitter, and timestamp recalibration is completed in the reconstructed frame; then, along the normal side of the reflection plane, the scene is slowly scanned near the reflection surface by a slider or gimbal at a low angle, and the observation frames that appear frame by frame are marked due to the superposition of the reflection surface and low-angle incidence, the staggered framing of the main view and the opposite side view, and the lag caused by exposure adaptation and supplementary light feedback.

[0045] A series of short-wave infrared specular ridges that drift slowly with the incident angle when a reflective surface is present and the incident angle varies slightly.

[0046] When a mirror path can be formed on the reflective surface and the main view and the opposite side view are framed in an alternating manner, observable symmetrical overlap of the real and virtual main view and the opposite side view appears as a continuous frame segment after reflection homography mapping, resulting in symmetrical overlap of the real and virtual images.

[0047] When the projections of the two view trajectories are converged by the reflective surface and the framing is staggered, synchronous overlap and delayed overlap of frames on the reflective surface occur.

[0048] In the near-infrared reflectance distinguishable and mirror path exists, there are consecutive frames in the main grayscale channel where the reflectance of fruit / leaf in the short-wave infrared channel is reversed compared to the mirror domain.

[0049] When the fill light or incident angle changes and overlaps the contour, a visible hysteresis occurs in the actual effective frame of the exposure adaptation and fill light feedback.

[0050] Then, the feature duration triggered by the root cause factor in this scene is calculated by defining the start and end frames of the labeled continuous frame segments:

[0051] The duration of the drift stabilization of the highlight ridge line is from the start of continuous drift of the highlight ridge line until the drift disappears; its origin is the drift of the normal direction of the short-wave infrared highlight caused by the slight change in the incident angle due to the reflection surface and the low angle of incidence.

[0052] The duration of virtual-real symmetry overlap: from the point where the virtual-real symmetry overlap is clearly rising until it falls back to the background level; its basis is that the mirror path and the body path induced by the reflection homography when the reflecting surface exists are briefly synchronized.

[0053] Hysteresis overlap clearing time: from the moment the main view and the opposite view begin to overlap on the reflective surface until they no longer overlap synchronously; its root cause is the convergence of the trajectory formed by the staggered framing of the main view and the opposite view on the reflective surface projection and the time lag.

[0054] Reflectance inversion regression time: from the time when the reflectance inversion occurs in the mirror domain within a short time window until it is restored to the normal contrast relationship; its basis is that the reflectance of the fruit and the background leaves in the near-infrared band can be distinguished and the superposition of the mirror paths causes a temporary reversal of the contrast relationship.

[0055] Time to cross a strong reflection zone: The time required for the line of sight to cross a strong reflection zone is calculated by the slow scanning and pausing frames of the slider or pan-tilt unit near the reflective surface; its basis is the strong zone dwell time determined by the spatial distribution of the reflective surface and the slow scanning / pausing.

[0056] Furthermore, the largest of the five calculated results is used as the time length baseline to cover the slowest links of specular ridge drift, virtual-real symmetry overlap, hysteresis overlap, reflectance reversal, and strong reflection zone crossing. If virtual-real symmetry overlap has not fallen back or hysteresis overlap has not been cleared at the end of the time length baseline, the time length baseline is extrapolated once to the first frame that simultaneously satisfies the fading of virtual-real symmetry overlap and the clearing of hysteresis overlap, based on slow scanning and the pause frame index. The extrapolated time length baseline is then determined as the closing time length of the shed reflection composite interference in this scene. At the same time, if a new exposure adaptive or supplementary lighting feedback effective frame appears within this time length baseline, the starting point of the relevant fading judgment is reset with the new frame, and the calibration is still completed by extrapolation only once.

[0057] The highlight ridges in the short-wave infrared channel are extracted as follows: the polarization channel difference is calculated as ΔP(x) = | (x)- (x)|, constructing a normalized specular indicator (x)=( (x)- (x)) / ( (x)+ (x)+ε); then perform a second-order Hessian calculation on the grayscale image, and let the minimum eigenvalue be (x)+ε). (x), then the ridge response (x)=max(0,- (x)), determine the set of ridge pixel as S={x| (x)≥ }

[0058] Subsequently, a sliding time window N1 with a time length equal to the closing time of the greenhouse reflection composite interference is set. Short-wave infrared high-light ridges are scanned and acquired in small steps along the normal side of the reflection plane with N1 as one period window. Then, the reflection plane equation, normal vector and reflection homography mapping matrix are obtained by robust fitting. Subsequently, a snapshot of the on-duty parameters of the reflection plane parameters, incident angle sequence, supplementary lighting state and camera attitude is established, consisting of the reflection plane equation, normal vector and reflection homography mapping matrix.

[0059] Next, the normal vector n (unit vector) of the reflection plane is substituted to construct an approximate homography mapping. ≈K·(I- )· As the initial value, for the matching point pair { , } Perform minimization of ∑i|| - · The actual reflection homography mapping matrix is ​​obtained by RANSAC robust estimation. And write it along with the reflection plane equation into the on-duty parameter snapshot.

[0060] Meanwhile, if the multi-band near-infrared imaging unit supports polarization switching or beam splitting, it records parallel / vertical polarization frame identifiers to distinguish between specular and diffuse reflection.

[0061] Using the aforementioned joint calibration parameter set as input, during the acquisition and alignment stage, the main camera, the opposite fixed-position camera, and the multi-band near-infrared unit are first connected to the edge image buffer array. Then, the frame number alignment is completed using a synchronous trigger line. Subsequently, timestamp recalibration is performed on the reconstructed frame. At the same time, polarization difference or normalized specular indicator is calculated in the short-wave infrared grayscale channel to generate a specular candidate mask, and all metadata is written with the frame.

[0062] The consistency score is composed of the following components: + 2·(1- )+ 3·ψ+ 4·exp(-∠(v, ) / ),in The polarization difference normalization value is given, ψ is the indicator of the 3D reconstructed point's location in front of the reflection plane (ψ=1 indicates front space), and v is... These represent the velocity vector directions of the main path and the mirror path, respectively. The angle normalization coefficient, , 2, 3, 4 represents the adjustable coefficient. If ≥ If ψ=1 and the region is not covered by polarized specular highlights, it is considered a candidate for the main body; the rest are marked as mirror images and stripped from the main chain.

[0063] Furthermore, based on the aforementioned aligned frame sequence and mirror candidate mask, the identification preprocessing first projects the key points of the hand and the fruit candidate in the main view onto the mirror domain via reflection homography, constructing a virtual-real symmetry consistency indicator. Then, it simultaneously calculates the geometric landing point physical solution with the epipolar constraint and sparse triangulation of the opposite side view, while continuously monitoring subtle changes in the incident angle and supplementary lighting state. When mirror candidates exist continuously and symmetry consistency increases, double solutions appear in the cross-view geometry, accompanied by subtle changes in the incident angle or supplementary lighting state, it triggers entry into the reflection decoupling channel.

[0064] Let x be the candidate point of the main view, and its mirror mapping be... = ·x, construct a consistent indicator (x)=exp(- )·NCC(P(x),P( ), where P(·) is the grayscale patch for the candidate region. The normalized scaling coefficient is NCC, which is the normalized cross-correlation function value. All parameters are written to the current shift parameter snapshot.

[0065] Using the trigger interval as the boundary, the reflection decoupling channel traces back to the mirror domain with the main view hand, fruit contact or approximation event as the main path to construct the mirror and ontology return chain. Then, the consistency score is calculated by combining the symmetric mapping residual, depth symbol and polarization difference intensity. Subsequently, the virtual and real allocation is performed according to the score: only candidates with physical properties, low polarization difference and direction consistent with the main path are retained as the ontology, and the rest are marked as mirrors and separated from the main chain. At the same time, the exposure / gain time axis of this segment is reconstructed with the slider / gimbal attitude and the fill light feedback event as anchor points to restore the true intensity sequence.

[0066] Subsequently, following the main path of the body to be mirrored, the trajectory layer then partitions and schedules the residual interference:

[0067] If it is determined that the mirror image is synchronized with the main body and the geometry falls on the mirror overlap segment behind the reflection plane, then the virtual viewpoint reprojection stripping is performed and the main chain is backfilled.

[0068] If the highlight ridge line slowly drifts with the incident angle and overlaps the highlight drift segment of the edge, then feature weighting and edge re-segmentation guided by highlight mask are performed;

[0069] If the incident angle lags behind the image intensity of the short time window after the attitude / lighting change, the timestamp micro-alignment and local exposure curve reconstruction are performed.

[0070] If a pseudo-closed segment is formed in the mirror domain only because of time proximity, then geometric consistency rejection and path disconnection are performed.

[0071] Using the above-annotated interference segment and main path as shielding conditions, the action judgment is confirmed if and only if the body domain has continuous contact, traction, separation and meets the geometric contact event closure. If only the mirror domain has fruit stem breakage or finger gripping posture change and the body domain has no contact, the action is rejected. If the mirror segment covers the key moment of the main path, the action time axis is backfilled with the corrected body strength and geometric sequence to avoid time drift and repeated counting.

[0072] Wherein, the hand-to-fruit distance is defined as d(t), and the normal displacement of the fruit handle is defined as Δn(t), then the behavior boundary rules are as follows:

[0073] Contact: d(t)≤ And duration ≥ ;

[0074] Traction: Under contact holding conditions, Δn(t) monotonically increases to ;

[0075] Separation: Connected graph from Connected becomes disconnected, and d(t) ≥ All threshold parameters Δ , , All parameters are written to the current shift's snapshot.

[0076] Based on the action timeline of the debiasing and the main path of the ontology, the data is merged and then pre-attribution reconstruction is performed: only ontology sequences with consistent labels, physical understanding, and closed paths are retained for model training and rule updates. At the same time, the reflection plane parameters, trigger frequency, incident angle / supplementary lighting status, and various stripped label statistics are written into the mirror governance area of ​​the shift parameter snapshot to form traceable feedback.

[0077] Finally, when scene statistics show that mirror interference occurs continuously in a specific orientation and incident angle range and the triggering frequency exceeds the handling threshold, the following actions are taken on the engineering side: adjust the slide rail travel boundary to avoid the main lobe of the reflection; adjust the supplementary light angle and polarizer direction to weaken the mirror component; add a depolarization component in front of the fixed camera on the opposite side to weaken the symmetry band; and set a low-weight band near the reflection plane in the path planning to reduce the dwell and retrace of low-angle incident light, thereby stabilizing short-wave infrared reflection, diffuse reflection discrimination boundary and reducing the probability of mirror triggering.

[0078] Furthermore, environmental sensing terminals supporting multi-modal sampling are symmetrically installed within the camera's field of view. These terminals contain multi-parameter probes for soil moisture, conductivity, and pH. The probe heads are helically embedded along the root extension axis of the target crop, and a waterproof polymer-sealed interface ensures long-term stable sampling. All probes employ high-impedance front-end circuitry connected to isolated operational amplifier channels, enabling the stable conversion of weak response electrical signals into standard analog voltage signals for output. For example, during fertilization, if the conductivity probe detects a step increase in amplitude on both sides of the target crop, it can be determined that the fertilizer coverage area meets the application requirements.

[0079] For agricultural operations involving water bodies, such as sprinkler irrigation and pesticide application, a high-sensitivity impeller-type flow sensor is pre-embedded in the sprinkler head bracket. This type of sensor encodes the spray start and stop events into discrete pulse signals through a built-in magnetic blade rotation speed counting circuit, and synchronously records them to the edge control unit via an SPI bus. When a continuous pulse sequence is detected after the sprinkler head is turned on, and the overlap time with the hand lifting action in the image frame meets a threshold condition, it can be determined as a valid spraying cycle.

[0080] Subsequently, in order to describe the correlation between operational behavior and environmental factors, a composite meteorological monitoring device with integrated data collection capabilities was installed above the crop canopy, integrating temperature, humidity, light intensity, and CO2 concentration measurement cores.

[0081] The temperature and humidity channel constructs a sampling curve based on the frequency shift principle of thermistors; the light sensor uses a programmable spectral segment selector to filter out interference from non-target wavelengths; the CO2 concentration channel is constructed through a non-dispersive infrared absorption module, and the output level is converted into ppm-level numerical frames after passing through a 12-bit ADC. These three signals are used together at the acquisition end to perform a moving average and fluctuation coefficient estimation within a time window, constructing the intensity distribution field of environmental factors during that period. For example, in facility agriculture, if a sudden change in CO2 concentration coincides with a reduction in crop shading and spraying behavior, it can be identified as an active ventilation operation event in a closed environment.

[0082] All analog output signals from the sensors are connected to an analog signal conditioning channel with high versatility configuration capabilities, consisting of a high-speed buffer stage, an adjustable gain amplification stage, and a single-pole RC filter stage connected in series. In the buffer stage, a rail-to-rail amplifier is used to increase the input impedance and alleviate load response. The amplification stage dynamically sets the gain ratio according to the sensor output amplitude. A low-pass filter suppresses interference signals exceeding the effective bandwidth limit to avoid abnormal fluctuations after analog-to-digital conversion. All conditioned signals are sampled in parallel by the ADC module inside the MCU main control unit. Sampling triggering is uniformly controlled by a hardware timer, enabling all channels to perform three stages—sampling start, sampling hold, and result reading—within a single sampling period, and then packaged and stored in the local data frame buffer in chronological order.

[0083] Finally, before each data acquisition cycle begins, the MCU automatically performs a short-circuit / open-circuit status scan on all input channels. It uses power-on feedforward detection and wiring self-test logic to determine if there are any abnormal conditions such as probe detachment, signal interruption, or level drift. During initial deployment, the main controller reads the nonlinear response curves and compensation parameters provided by various sensors and constructs a linear transformation parameter set based on a minimum residual fitting algorithm. This parameter set is then permanently written into the local Flash memory as a lifecycle calibration benchmark.

[0084] In the data preprocessing and fusion module, after acquiring the multimodal raw data, frame-by-frame preprocessing is performed on the image sequence. Specifically, bilateral filtering is prioritized in the image channels to denoise and preserve edges in edge regions, and the Retinex enhancement model is activated when the dynamic distribution of light and dark areas is uneven to improve the discernibility of the subject in high-contrast scenes. Subsequently, grayscale normalization and exposure time reestimation are performed on the original images, and images with insufficient inter-frame variation are discarded to compress inefficient computational resources.

[0085] The preprocessing link is implemented in an FPGA acceleration array deployed at the edge node, supporting parallel processing of image partitions and synchronous triggering of multiple channels.

[0086] Simultaneously, numerical data from various environmental and crop status sensors are input into a unified formatted channel. All analog signals are digitized by the ADC module, then sequentially pass through a moving average filter to eliminate periodic pulse disturbances, employ first-order difference to identify edge transition points, and finally perform dynamic interpolation reconstruction based on cubic splines on time series with discontinuities to fill in the gaps in the sampling time slots. For example, in some facilities where spraying activities are affected by signal obstruction leading to missing CO2 concentration signal samples, the continuous trend can be restored through the reconstruction model, and response weights can be applied to attenuate abnormal abrupt changes.

[0087] Furthermore, to achieve frame alignment between images and sensor information in the temporal dimension, a multi-source synchronization mechanism based on the main image frame sequence is constructed. Specifically, a hardware timestamp locking mechanism is used to mark the acquisition time of each image frame and each group of sensor sampling values, and a benchmark queue window is established. Image frames with a time difference less than the synchronization threshold are bound one-to-one with the sensor group. If there is data source delay or packet loss, a zero-order hold compensation term is automatically inserted or a valid null value label is set. For example, in drip irrigation behavior monitoring, if the soil conductivity value is delayed in updating while the image frame captures the starting frame of water splashing, the nearest neighbor hold method is used to ensure that the temporal consistency is not disrupted.

[0088] After completing time alignment, a cross-modal fusion vector is constructed for each frame of bound data. The fusion features include image region texture tensor, RGB color statistical features, instantaneous values ​​of sensor values, rate of change, weighted environmental background factors, etc., and are compressed to a unified scale space through normalization mapping.

[0089] In the sample library and indicator system module, to achieve stable recognition of various agricultural behaviors under complex environments, the image and sensor fusion vectors are first input into a dual-channel SlowFast feature extraction network. The Slow path captures the overall rhythm of the operation at a lower frame rate, focusing on the temporal scale distribution of operational actions and the semantic transfer process between keyframes; the Fast path performs high-frequency sampling of fine-grained temporal changes at a high frame rate, enhancing the ability to recognize short-term dynamic features at the start and end of the behavior.

[0090] The two paths introduce feature cross-layer connection structures between multiple resolution layers and introduce dual-scale residual convolution modules in the time axis direction, so that the model can take into account both long-term operational patterns and short-term action fluctuations.

[0091] To enhance the model's adaptability to differences in crop varieties, growth stages, and operational rhythms, an SMC-ECA attention module is introduced on top of SlowFast to dynamically reweight the modal saliency of each dimension in the input vector. Specifically, the SMC module constructs channel complementarity relationships within a sliding window based on the contextual features of the current input frame and assigns greater weight to highly correlated modalities based on spatial variability and behavioral co-occurrence probability. The ECA module captures fine-grained collaborative changes in neighboring channels through one-dimensional convolutional kernels to suppress redundant information interference and improve the model's focusing ability under multimodal heterogeneous inputs. For example, in fruit harvesting, the image modality may lose target clarity due to backlighting conditions. In this case, the attention weight of the fruit stalk stress / temperature and humidity change signals in the sensing modality will increase significantly, ensuring that behavioral judgment is not affected by the absence of the main visual channel.

[0092] Among them, the input multimodal fusion vector =[Texture Tensor, RGB Statistics, Sensor Values, Rate of Change, Environmental Factors]; The SMC module calculates the channel complementarity matrix. Perform modal reassignment: = · The ECA module performs one-dimensional convolution to extract collaborative relationships. =Conv1d(gap( Generate attention coefficient ∈(0,1). The total loss function is L= +λ1· +λ2· During the inference phase, the output sequence is smoothed using GRU, and then weighted smoothed using EMA to generate the final classification labels. =argmaxEMA(GRU(p1:t)).

[0093] Furthermore, the aforementioned deep feature sequences are input into a gated recurrent unit (GRU) network, and the output of the current frame category is dynamically adjusted in conjunction with the prior state of historical behaviors. Each behavior category output node is configured with pre- and post-warm-up sections and a delayed release window to avoid mis-segmentation of behaviors due to inter-frame fluctuations.

[0094] To further improve the model's discrimination accuracy in scenarios with weak feature behaviors (such as flower thinning and topdressing), a multi-stage candidate nomination mechanism is adopted to construct a candidate set screening and behavior voting strategy. First, high-confidence candidate behavior labels and their occurrence time periods are extracted from the basic output of each frame; then, short-time dynamic time warping (DTW) matching is performed by constructing a behavior similarity matrix, and weighted moving average filtering is applied to the confidence sequence; the final output behavior judgment result is the maximum vote label of the candidate behavior set.

[0095] Let the set of candidate behaviors be {( )},in As candidate tags, For confidence level, For the corresponding frame segment, temporal similarity weights are obtained through Dynamic Time Warping (DTW) matching. Construct a weighted voting score S(y) = The final behavior label is This value is written to the current identification frame segment.

[0096] Each identified behavior segment, along with its image frame, sensor time series value, behavior category label, and working environment parameters, is packaged together and written into a standardized database according to a four-level classification structure: crop type, behavior type, environmental conditions, and working result.

[0097] The database uses a multi-level index structure to organize sample paths, which facilitates fine-grained training, rapid retrieval, and sample tracing of the behavior classification model.

[0098] Meanwhile, to achieve quantifiable evaluation and standardized judgment of behavioral quality, an indicator system framework is constructed around various operational behaviors. The indicator system has a three-layer structure:

[0099] The first layer is the behavioral structure integrity index, which mainly evaluates whether the start-process-end segments of the operation are complete. It is often presented in the form of frame ratio, action boundary continuity, etc.

[0100] The second layer is the rhythm coordination index, which is used to measure the temporal uniformity of the operation, the range of frequency fluctuations, and the level of rhythm drift.

[0101] The third layer is the environmental fit index, which measures the degree of matching between the environment and the crop stage when the behavior occurs.

[0102] All indicators are calculated based on the metadata of the behavior fragments and are automatically classified into the indicator distribution range of the corresponding behavior type, so as to dynamically update the evaluation boundary and operation threshold.

[0103] Structural integrity index: =( ) / ∈(0,1]; Rhythmic Coordination Index: ρ=1-CV({ }), CV is the coefficient of variation of the inter-frame interval; environmental fit index: η=sim( ), where sim is the normalized similarity (such as cosine similarity or Pearson correlation). After normalizing all indicators, they are written into the corresponding behavior label.

[0104] Furthermore, a sample screening and template extraction mechanism is preset in the sample library module. Based on the performance indicators of historical behavior segments, a screening threshold is set, retaining only high-quality segments with complete structure, stable rhythm, and good environmental adaptability as candidate behavior templates. Candidate segments are grouped into several typical templates through feature clustering methods. Each type of template is bound to the source crop, execution stage, and standard operation video stream for rapid distribution and comparison when deploying in new environments.

[0105] In addition, to adapt to regional differences, variety updates and behavioral evolution trends, dynamic adjustment of indicator boundaries and sample drift perception are adopted. By analyzing the indicator distribution density, skewness and abnormal clustering of continuous input behavioral segments, it automatically determines whether existing behavioral templates need to be loosened, supplemented or abolished.

[0106] In each sample record, the data acquisition device number, the identification model version number, and the corresponding processing flow ID are recorded simultaneously to ensure that any historical sample has a complete traceability path, which can be used for behavioral anomaly review, equipment difference analysis, and model tracing.

[0107] In the visualization traceability and decision support module, to achieve visualized tracking and decision support response for agricultural production behaviors, a unified data pipeline is constructed based on the sample library and recognition result stream, spanning behavior recognition, indicator evaluation, anomaly analysis, and behavior backtracking. Specifically, at the first moment after the recognition output is generated, the main image frame, sensor value sequence curve, and environmental indicator time history corresponding to that behavior segment are automatically captured, and the data is bound to the image using a frame sequence and line chart linkage method to generate a nested time-series behavior trajectory layer.

[0108] The trajectory layer consists of an image frame main window, a behavior timeline, and synchronous multi-physical quantity curves, supporting behavior tracing and image frame skipping operations based on mouse drag.

[0109] Subsequently, in order to form a comparative view of indicators for multiple batches of operation processes, a behavior evaluation summary table is constructed after each behavior is identified. This summary table uses behavior category as the main axis and structural integrity index, rhythm synchronization index and environmental fit index as the horizontal axis to construct a scoring matrix. The behavior reference template matching degree and the distribution of current operation abnormal labels are added for comprehensive evaluation of results.

[0110] All indicators are displayed side by side as the deviation between the single score and the historical average, with abnormal items highlighted. Filtering and redrawing are supported by operation batch, time period, or crop stage to help users determine the trend of operational stability.

[0111] In terms of tracing the source of behavioral anomalies and exporting results, an automatic trajectory callback mechanism based on behavioral sequence triggering is constructed. When a user clicks on any scoring item or behavioral segment in the indicator summary, the original data package of that segment will be retrieved immediately, including the main image frame, recognition path, intermediate layer feature map, and sensor data stream, and uniformly encapsulated in the backtracking window for presentation. It supports automatic playback in chronological order or manual frame-by-frame progression. At the same time, the system provides metadata fields such as behavior occurrence time, device ID, model version, and behavior comparison template number, allowing users to manually annotate behavioral events or export anomaly report files.

[0112] Furthermore, in multi-site deployment scenarios, to support cross-site behavior aggregation and unified analysis, a behavior aggregation engine and an operation hotspot monitoring module are built. Based on behavior recognition logs and device location codes in the database, a heat map of behavior events is automatically drawn, marking high-frequency operation points and abnormal cluster areas. At the same time, users can optimize operation paths and adjust operation intensity based on a visual view.

[0113] Finally, when users make comprehensive decisions on the platform, the platform's inference engine invokes strategy rules and optimization suggestion libraries to provide policy recommendations, personnel scheduling strategies, or traceability solutions based on the current identified behaviors and environmental conditions. If the user selects a target area and crop type, the platform will summarize the frequency of behavior identification, operational standardization rate, and abnormal fluctuation trends in that area over the recent period, and output actionable suggestion text in conjunction with historical backtracking logs. For example, if the current fruit thinning operation stage is experiencing high frequency of rhythm drift, it is recommended to prioritize experienced operators; simultaneously, the suggestion text can be synchronously sent to the agricultural machinery collaborative platform, automatic broadcasting system, or expert remote diagnostic terminal via interface to achieve a multi-source linkage behavior management closed loop.

[0114] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0115] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0116] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0119] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target recognition system based on multimodal image analysis, characterized in that: include: The system includes a multimodal data acquisition module, a data preprocessing and fusion module, a sample library and indicator system module, and a visualization traceability and decision support module, with signal connections between the modules. The multimodal data acquisition module is used to slowly scan along the normal side of the reflection plane by a slide rail or pan-tilt unit at a low angle, label continuous frame segments frame by frame and calculate the time length baseline. Based on the time length baseline, the closing time length of the greenhouse reflection composite interference is determined, and the sliding time window is set with the closing time length of the greenhouse reflection composite interference. The reflection plane parameters and mirror candidate masks are constructed by the reflection plane equation, normal vector and reflection homography mapping matrix, and the body and mirror are classified and labeled according to the consistency score. The data preprocessing and fusion module is used to construct a multi-source synchronous binding relationship based on the main image frame through image filtering and enhancement, sensor signal reconstruction and formatting, and output a fusion vector; The sample library and indicator system module is used to extract temporal features by inputting the fusion vector into the dual-channel SlowFast network and performing modality reweighting, and then combine it with GRU to output behavior classification labels; The visualization traceability and decision support module is used to generate behavior trajectory layers and summary tables by building a data pipeline that runs through the identification results and indicator evaluation, and output scheduling suggestions and disposal plans based on the inference engine.

2. The target recognition system based on multimodal image analysis according to claim 1, characterized in that: A variable focal length high-resolution CMOS image sensing module was deployed, and the elevation angle was calibrated using a ground reference point and a sensing cone. Then, a multi-band near-infrared imaging unit was installed and frame-level matching was performed. Then, the variable focal length high-resolution CMOS image sensing module and the multi-band near-infrared imaging unit are connected to the edge image buffer array. After high-pass filtering and timestamp recalibration, a frame-level binding relationship is established between the main image frame sequence and the frame-level matching result of the multi-band near-infrared imaging unit. A gray-scale channel image is constructed based on the multi-band near-infrared imaging unit, and an exposure equalization reference window is set in the main stem area of ​​the crop to determine the action criteria.

3. The target recognition system based on multimodal image analysis according to claim 1, characterized in that: During the deployment phase, geometric and optical joint calibration is performed. After all optoelectronic devices are connected to the edge image buffer array, high-pass band filtering and timestamp recalibration are performed. The slide rail or gimbal slowly scans along the normal side of the reflection plane at a low angle. Frame by frame, continuous frame segments appear due to the superposition of the reflection surface and low-angle incidence, the staggered framing of the main view and the opposite side view, and the lag caused by exposure adaptation and supplementary light feedback. Then, the corresponding feature durations are calculated based on the marked start and end frames, and the largest of the feature durations is taken as the time length baseline. When the virtual and real symmetrical overlaps do not fall back or the hysteresis overlaps are not cleared at the end of the time length baseline, the frame is extrapolated once according to the slow scan and the pause frame index until the frame that simultaneously satisfies the disappearance of virtual and real symmetrical overlaps and the clearing of hysteresis overlaps is reached for the first time. The extrapolated time length baseline is then determined as the closing time length of the shed reflection composite interference.

4. The target recognition system based on multimodal image analysis according to claim 3, characterized in that; By setting the sliding time window N1 based on the closing time of the chamber reflection composite interference, the short-wave infrared high-light ridge is scanned in small steps along the normal side of the reflection plane. The reflection plane equation, normal vector and reflection homography mapping matrix are constructed, and a snapshot of the on-duty parameters including the reflection plane equation, normal vector, reflection homography mapping matrix, incident angle sequence, supplementary lighting state and camera attitude is established.

5. The target recognition system based on multimodal image analysis according to claim 4, characterized in that: During the acquisition and alignment phase, the main camera module, the opposite fixed-position camera, and the multi-band near-infrared imaging unit are all connected to the edge image buffer array. The main camera module forms the main view, the opposite fixed-position camera forms the opposite side view, and the multi-band near-infrared imaging unit is used to construct short-wave infrared grayscale channel images. The frame number alignment and timestamp recalibration are completed through synchronous triggering, and the polarization difference or normalized specular indicator is calculated in the short-wave infrared grayscale channel to generate a candidate mirror mask.

6. The target recognition system based on multimodal image analysis according to claim 5, characterized in that: Normalized cross-correlation and spatial position consistency judgment are performed on the main view candidate and the projection point of the mirror domain to form a consistency score. If the spatial judgment is true and the target area is not covered by specular highlights, it is marked as the main body candidate. Otherwise, it is stripped as the mirror and the exposure and gain time axis of this stage is reconstructed after the reflection decoupling channel is triggered.

7. The target recognition system based on multimodal image analysis according to claim 1, characterized in that: Environmental sensing terminals with multimodal sampling capabilities are symmetrically installed in the visible area of ​​the camera equipment to collect signals of soil moisture, conductivity, pH, spraying flow rate and meteorological factors, and are uniformly connected to the main control ADC module for synchronous sampling through an analog signal conditioning channel; Before sampling, a channel status scan is performed. During the initial deployment, response curves of various sensors are read and compensation parameters are established. These parameters are written into Flash to form a calibration benchmark and to construct a correlation description between operational behavior and environmental factors.

8. The target recognition system based on multimodal image analysis according to claim 1, characterized in that; The acquired image sequences are subjected to frame-by-frame bilateral filtering and Retinex enhancement, and the image partitioning is performed in parallel on the edge node FPGA array. The outputs of various environmental and crop sensors are sampled by ADC and then reconstructed by moving average, first-order difference and cubic spline interpolation in sequence to eliminate signal discontinuities and abrupt disturbances. Construct a multi-source synchronization binding relationship, perform frame alignment between image frames and sensor group data, and insert zero-order hold or null label compensation for delay sources.

9. The target recognition system based on multimodal image analysis according to claim 1, characterized in that: The fused vector is input into a dual-channel SlowFast feature extraction network. The Slow path captures the operation rhythm and keyframe semantic transfer, while the Fast path extracts high-frequency short-term dynamic features. Based on this, each modality dimension is dynamically reweighted. Subsequently, the deep feature sequence is smoothed using a GRU network, and pre- and post-warm-up zones and delay windows are set to prevent erroneous cutting.

10. The target recognition system based on multimodal image analysis according to claim 1, characterized in that: A visualized data pipeline is constructed that runs through the identification output, indicator evaluation, and behavior backtracking. The main frame of the image, the sequence curve of the sensor value, the time history of the environmental indicator, and the indicator score are associated to generate a nested time-series behavior trajectory layer and a behavior evaluation summary table. An operation hotspot monitoring module and a behavior event heat map are constructed, and policy recommendations, personnel scheduling strategies, or behavior handling plans are output based on the platform's inference engine.

Images