Pineapple transplanting machine operation monitoring method and system based on image analysis

Abstract

The present application provides a kind of based on image analysis's pineapple transplanting machine operation monitoring method and system, it is related to pineapple transplanting machine technical field, the present application is by constructing transplanting actuator full-phase no-load standard image benchmark library, with running phase signal time sequence correlation formation operation state pair dynamic image stream;Difference image is obtained by phase matching, image preprocessing and interframe difference, accurately distinguish seedling and mechanism background;Based on difference image, determine whether seedling exists, extract seedling contour and calculate the spatial offset of geometric center and seedling center of projection, construct offset sequence;Abnormal proportion is counted using sliding window, realize stable abnormal early warning, the present application can eliminate mechanical structure and field environment interference, realize seedling existence, position offset and full cycle accurate monitoring of card stagnation and miss transplanting, can effectively improve pineapple transplanting operation precision and survival rate, suitable for intelligent monitoring of various pineapple transplanting machines.

Description

Technical Field

[0001] This invention relates to the field of pineapple transplanter technology, specifically to a method and system for monitoring pineapple transplanter operations based on image analysis. Background Technology

[0002] When pineapple transplanters operate continuously in the field, seedlings are prone to abnormalities such as clamping deviation, jamming, folding, missed planting, and seedling placement deviation due to factors such as differences in seedling shape, mechanical vibration, and variable light. These abnormalities directly reduce the survival rate of transplanted seedlings and the quality of the operation.

[0003] Existing monitoring technologies mostly use photoelectric sensors to detect the duration of occlusion or the presence or absence of simple images. They can only determine the presence of seedlings, but cannot obtain information on the posture, position, and offset of the seedlings within the actuator. When seedlings are slightly offset, partially stuck, or folded and deformed but do not completely obscure the sensor, missed detections are likely to occur. When field vibrations or weed interference cause momentary occlusion exceeding the time limit, false detections are likely to occur. During continuous operation of machinery, it is impossible to guarantee that the real-time image and the machine position are accurately matched, making it difficult to remove background interference and resulting in poor recognition stability. Therefore, there is an urgent need in this field for a monitoring method for pineapple transplanter operations that can accurately match phases, eliminate background interference, quantify seedling offset, and achieve stable early warning based on time-series statistics.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for monitoring pineapple transplanter operations based on image analysis, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for monitoring pineapple transplanter operations based on image analysis, comprising the following steps: S1: Construct a spatial feature benchmark library containing transplanting actuators. The spatial feature benchmark library presets no-load standard images of each operating phase of the actuator and sets the corresponding operating phase intervals. S2: Receive the dynamic image stream of the transplanting actuator during the transplanting operation in real time, and synchronously acquire the operating phase signal of the transplanting actuator corresponding to the dynamic image stream. Then, perform time-series logical association between each frame image sequence in the dynamic image stream and the operating phase signal to form a work state pair corresponding to each operating phase. S3: By matching the running phase signal in each job state pair with the running phase interval in the spatial feature reference library, determine the corresponding no-load standard image, preprocess the image in each job state pair, and perform background difference processing with the corresponding no-load standard image to obtain the difference image. S4: Based on the difference image, determine whether there are seedlings in the transplanting actuator corresponding to each work state pair; if there are seedlings, extract the seedling image area in each work state pair, calculate its geometric center position, preset the seedling placement center position according to the transplanting actuator parameters, calculate the spatial offset between the seedling geometric center and the seedling placement center, and form an offset sequence related to the running phase signal. S5: Set a preset offset threshold, extract values ​​in the offset sequence whose offset is greater than the offset threshold, and mark them as potential abnormal states; set a preset sliding window with a duration adapted to a complete operating cycle of the transplanting actuator, and count the proportion of potential abnormal states in each sliding window; if the proportion of potential abnormal states in the sliding window exceeds a preset proportion threshold, an abnormal warning signal is output.

[0007] Furthermore, the operating phase interval is a continuous range of phase values, and the operating phase intervals do not overlap with each other.

[0008] Furthermore, when constructing the spatial feature benchmark library, the transplanting actuator needs to be run under no-load conditions in all phases, and images under each operating phase need to be collected. After denoising and grayscale processing, these images are used as no-load standard images, and the phase span of each operating phase interval does not exceed 5°.

[0009] Furthermore, the matching of the operating phase signal and the operating phase interval adopts the interval inclusion determination method: the specific value of the operating phase signal in the operation status is compared with the boundary value of each operating phase interval in the spatial feature reference library. If the value falls within the range of a certain operating phase interval, the no-load standard image corresponding to that operating phase interval is called as the background reference image.

[0010] Furthermore, the preprocessing steps for the images in each job state pair are consistent with the preprocessing steps for the unloaded standard image, specifically including denoising, grayscale conversion, image enhancement, and edge smoothing steps. The background differentiation processing adopts the inter-frame difference method, calling the unloaded standard image that matches the phase of the current job state pair and has undergone the same preprocessing, calculating the grayscale difference between the current preprocessed image and the unloaded standard image pixel by pixel, and marking the pixels with grayscale differences greater than the preset grayscale threshold as foreground pixels to form a difference image; the preprocessing process is synchronously associated with the running phase signal in S2.

[0011] Furthermore, the specific method for determining whether there are seedlings based on the difference image is as follows: the difference image is binarized, and foreground pixels with a grayscale difference greater than a preset grayscale threshold are set to 255, while background pixels with a grayscale difference less than or equal to the preset grayscale threshold are set to 0, thus achieving clear separation between the foreground and the background; then, the total number of foreground pixels in the binarized difference image is counted. The preset pixel threshold is set according to the minimum projected area of ​​the pineapple seedling. If the total number of foreground pixels is greater than the preset pixel threshold, it is determined that there are seedlings in the transplanting actuator; otherwise, it is determined that there are no seedlings and marked as a seedling absence anomaly, and the phase signal and corresponding image frame of this operation status pair are recorded simultaneously.

[0012] Furthermore, when seedlings are determined to be present, a contour extraction algorithm is used to extract the complete contour of the seedling image region based on the binarized difference image. Scattered pixels at the contour edges are removed to ensure contour accuracy. Then, based on the coordinate information of all pixels on the contour, the geometric center position of the seedling image region is calculated using the centroid calculation formula. Based on the center coordinates of the seedling delivery channel of the transplanting actuator, the coordinate parameters of the seedling delivery center position are preset. The spatial offset between the geometric center of the seedling and the delivery center is calculated using a spatial coordinate calculation method. The spatial offsets corresponding to each operation state are arranged sequentially according to the time sequence of the running phase signals to form an offset sequence that corresponds one-to-one with the running phase signals.

[0013] Furthermore, the offset sequence is traversed, and values ​​with offsets greater than the offset threshold are extracted and marked as potential abnormal states. Then, a sliding window of preset duration is set, with the preset duration of the sliding window matching 1-2 complete operating cycles of the transplanting actuator to ensure that it can cover one complete seedling placement process of the actuator.

[0014] Furthermore, when an abnormal warning signal is output, the corresponding running phase, offset value, and corresponding image frame are recorded simultaneously, and the abnormal information is stored in the local database and sent to the transplanter control system to control the transplanter to reduce its running speed or suspend operation.

[0015] Furthermore, the present invention also provides an image analysis-based pineapple transplanter operation monitoring system for executing the above-mentioned monitoring method, characterized in that it includes: Benchmark library construction module: Constructs a spatial feature benchmark library containing transplanted actuators. The spatial feature benchmark library presets no-load standard images of the actuators under each operating phase and sets the corresponding operating phase intervals. Data synchronization and association module: Receives dynamic image streams from the transplanting actuator during the transplanting operation in real time, and synchronously acquires the operating phase signals of the transplanting actuator corresponding to the dynamic image stream. It then performs time-series logical association between each frame of the image stream and the operating phase signals to form operation status pairs corresponding to each operating phase. Image difference processing module: By matching the running phase signal in each job state pair with the running phase interval in the spatial feature reference library, the corresponding no-load standard image is determined. The image in each job state pair is preprocessed and background difference processing is performed with the corresponding no-load standard image to obtain the difference image. Seedling offset detection module: Based on the difference image, it determines whether there are seedlings in the transplanting actuator corresponding to each operation state pair; if there are seedlings, it extracts the seedling image area in each operation state pair, calculates its geometric center position, presets the seedling placement center position according to the transplanting actuator parameters, calculates the spatial offset between the seedling geometric center and the seedling placement center, and forms an offset sequence related to the running phase signal; Anomaly warning and judgment module: a preset offset threshold is set, and values ​​with offset values ​​greater than the offset threshold in the offset sequence are extracted and marked as potential abnormal states; a preset sliding window with a duration adapted to a complete operating cycle of the transfer actuator is used to count the proportion of potential abnormal states in each sliding window; if the proportion of potential abnormal states in the sliding window exceeds a preset proportion threshold, an anomaly warning signal is output.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a full-phase no-load standard image benchmark library for transplanting actuators, achieving unique matching of "operating phase - no-load image". Combined with inter-frame differential to eliminate interference from the actuator's own structure, it can stably extract seedling areas even in complex environments such as strong light, weak light, dust, and vibration, with high recognition accuracy. By strictly binding the dynamic image stream with the operating phase signal in time sequence, it forms an "image-phase" operation state pair, ensuring that each frame corresponds to a unique mechanical position, realizing continuous monitoring of the entire process and all phases from seedling picking and transportation to seedling placement. Based on the difference image, it realizes the determination of seedling presence and absence, and further quantifies the spatial offset between the seedling and the placement center, achieving a technical upgrade from "presence and absence judgment" to "precise offset detection". At the same time, it adopts a constant judgment mechanism of "offset threshold + sliding window + proportion judgment" to filter out accidental anomalies caused by instantaneous interference, and only outputs warnings for continuous and systematic faults, making the warnings more stable and reliable, and avoiding frequent false triggers that affect operation efficiency. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2This is a schematic diagram of the overall system structure of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Example: Please see Figure 1 The present invention provides a technical solution: A method for monitoring pineapple transplanter operations based on image analysis, comprising the following steps: S1: Construct a spatial feature benchmark library containing transplanting actuators. The spatial feature benchmark library presets no-load standard images of each operating phase of the actuator and sets the corresponding operating phase intervals. The operating phase intervals are continuous ranges of phase values, and each operating phase interval does not overlap. When constructing the spatial feature benchmark library, the transplanting actuator needs to be operated without load in all phases. Images under each operating phase are collected, and after denoising and grayscale processing, they are used as no-load standard images. No-load refers to the unloaded state in which the transplanting actuator is not holding or transporting pineapple seedlings. The transplanting actuator needs to be controlled to operate without load in all phases. The complete operation process of the actuator from seedling picking and transporting to seedling placement is covered. Corresponding images are collected under each operating phase, and after denoising and grayscale processing, they are used as no-load standard images and stored in the benchmark library to ensure that there is a corresponding standard image reference for each subsequent operating phase. The core objective of this step is to establish a clean background image library that strictly corresponds to the mechanical position. This library will be used in subsequent steps to accurately remove image interference from the transplanting actuator's own structure using a differential method, thereby highlighting the seedling target.

[0021] In a preferred embodiment, the specific implementation process for constructing the spatial feature benchmark library is as follows: In the debugging workshop, an industrial camera is fixedly installed in a position that can clearly and completely capture the entire working cycle of the transplanting actuator, such as key parts like the clamping arm, conveyor chain, and seedling inlet. Uniform and stable lighting conditions are ensured to simulate typical field lighting conditions. A high-precision rotary encoder is installed on the drive shaft of the transplanting actuator to acquire the actuator's operating phase signal in real time. This phase signal, with an angle value from 0° to 360°, represents the real-time position of the mechanism within a complete working cycle. Camera parameters are adjusted to ensure the dynamic range of the captured image can adapt to the mechanism's movement, avoiding motion blur and ensuring a clear outline of the mechanism in the image. This facilitates subsequent full-phase no-load image acquisition, image preprocessing and phase matching, and the construction of a spatial feature reference library. No-load image acquisition includes: controlling the pineapple transplanter to operate in an unloaded state without clamping or conveying any pineapple seedlings, aiming to ensure that the acquired images only contain the structural features of the transplanting actuator itself, without any seedlings or foreign objects. The transplanter is then started and cyclically run at its rated operating speed. The encoder and camera are synchronously triggered to ensure that the precise phase angle value fed back by the encoder is recorded simultaneously for each acquired image frame. The acquisition must cover a complete work cycle (0° to 360°) of the transplanting actuator, from its initial position, through conveying and positioning, to the seedling placement point, and back to the initial position. Image preprocessing and phase pairing include: performing a unified preprocessing procedure on all acquired raw images, including: using a Gaussian filtering algorithm to reduce image noise, using a weighted averaging method to convert color images into grayscale images, and unifying the benchmark for subsequent processing. After preprocessing, each processed grayscale image is bound to its unique phase angle value recorded synchronously during acquisition, forming a raw data pair of "phase value-standard image".

[0022] The construction of the spatial feature benchmark library includes: based on the transplanter's working cycle and monitoring accuracy requirements, setting the span of each phase interval, dividing a 360° working cycle into 5° intervals, resulting in 72 operating phase intervals. Each interval corresponds to a small, relatively fixed positional state of the mechanism. For each operating phase interval, all preprocessed images contained within it are averaged at the pixel level to generate a smooth, no-load standard image that best represents the mechanism's state in that interval. Finally, each operating phase interval corresponds to a unique no-load standard image. The correspondence between all "operating phase intervals and no-load standard images" is stored in a database or a specific file, forming the spatial feature benchmark library.

[0023] S2: Receive the dynamic image stream of the transplanting actuator during the transplanting operation in real time, and synchronously acquire the operating phase signal of the transplanting actuator corresponding to the dynamic image stream. Then, perform time-series logical association between each frame image sequence in the dynamic image stream and the operating phase signal to form a work state pair corresponding to each operating phase. The core objective of this step is to acquire real-time operational status data of the transplanting actuator during the transplanting process, establish an "image-phase" correspondence, and provide a unified and consistent foundation of data for subsequent phase matching, background differentiation processing, and anomaly detection. The dynamic image stream refers to a continuous sequence of images, including those of the transplanting actuator, acquired in real-time during the transplanting operation. The frame rate must meet the operating speed requirements of the actuator to ensure complete capture of its status at each operational phase. Furthermore, the acquisition range must cover the entire process of clamping, conveying, and seedling placement, ensuring that each operational phase corresponds to a clear image frame. The operational phase signal refers to a signal that is synchronized with the dynamic image stream and reflects the real-time operational position of the transplanting actuator. This signal is acquired in real-time by an encoder installed on the drive shaft of the actuator, with the acquisition frequency synchronized with the frame rate of the dynamic image stream to ensure that each image frame corresponds to a unique operational phase signal. By associating the images, the system can find the unloaded standard image at the same phase position in the spatial feature benchmark library constructed in step S1 for each real-time image. This enables subsequent steps to accurately and dynamically remove the fixed background texture generated by the mechanical structure itself at the corresponding position, thus clearly highlighting the seedling as the foreground target even in complex environments. This is the core solution to the problem of difficult background interference removal. S3: By matching the running phase signal in each job state pair with the running phase interval in the spatial feature reference library, determine the corresponding no-load standard image, preprocess the image in each job state pair, and perform background difference processing with the corresponding no-load standard image to obtain the difference image. The matching of the running phase signal and the running phase interval adopts the interval inclusion determination method: the specific value of the running phase signal in the operation status is compared with the boundary value of each running phase interval in the spatial feature reference library. If the value falls within the range of a certain running phase interval, the no-load standard image corresponding to that running phase interval is called as the background reference image. The specific image preprocessing operations are consistent with the preprocessing operations of the unloaded standard image in S1, with the aim of ensuring that the grayscale reference of the preprocessed image is consistent with that of the unloaded standard image. Specifically, this includes sequential denoising, grayscale conversion, image enhancement, and edge smoothing: In this embodiment, denoising uses a Gaussian filtering algorithm with a filter kernel size of 3×3; grayscale conversion uses a weighted average method, calculating grayscale values ​​according to RGB channel weights of 0.299, 0.587, and 0.114; image enhancement uses a histogram equalization algorithm to improve image grayscale contrast; and edge smoothing uses a median filtering algorithm with a filter kernel size of 3×3. The size is set to 5×5; the background differentiation processing adopts the inter-frame difference method, specifically: call the unloaded standard image in S1 that matches the phase of the current operation state (this image has completed the same preprocessing as the image of the current operation state), calculate the gray level difference between the preprocessed image of the current operation state and the unloaded standard image pixel by pixel, and mark the pixels with gray level difference greater than the preset gray level threshold as foreground pixels to form a difference image; and during the preprocessing process, the running phase signal corresponding to the current image in S2 is called synchronously, and the parameters of each step of the preprocessing are recorded to ensure the temporal consistency with the seedling area extraction and geometric center calculation in the subsequent S4.

[0024] The grayscale threshold mentioned above is set to 20-50. To further enhance the practicality of the method, it can be dynamically adjusted according to the light intensity of the dynamic image stream in S2. Since the light intensity of open-air light at noon on a sunny day is about 10,000-20,000 lux, and the light intensity of open-air light on a cloudy day is about 3,000-5,000 lux, when the light intensity is strong, that is, when the light intensity is greater than or equal to 5,000 lux, the grayscale threshold is increased to 35-50 to avoid grayscale misjudgment caused by strong light reflection. When the light intensity is weak, that is, when the light intensity is less than or equal to 5,000 lux, the grayscale threshold is decreased to 20-35 to ensure that the seedling area can be effectively identified. Moreover, the grayscale threshold adjustment process is synchronously associated with the running phase signal of the current operation status pair. A uniform grayscale threshold is used in the same running phase range to ensure the consistency of the difference image. Pineapple transplanters operate in open fields under complex and variable lighting conditions. In strong light, the metal transplanting mechanism is prone to high-level reflections, creating bright spots similar to the seedlings in the differential image and mistakenly increasing foreground pixels. In weak light, the grayscale contrast between the seedlings and the background decreases, resulting in a smaller grayscale difference in the seedling area after differential processing. Fixed thresholds cannot simultaneously adapt to both extreme conditions; dynamic adjustment strategies can mitigate the negative impact of lighting variations on image segmentation to some extent. To verify the effectiveness of the dynamic grayscale threshold adjustment scheme, this application conducted comparative tests during the development phase. Multiple rounds of transplanting operations were carried out under two typical environments: sunny day with strong light (approximately 15,000 lux) and cloudy day with weak light (approximately 4,000 lux). The same set of acquired operation status image sequences were processed using both the fixed threshold method and the aforementioned dynamic threshold method. Table 1: Performance Comparison of Different Gray-Scale Threshold Settings As shown in the table above, the scheme using dynamic adjustment of grayscale thresholds based on light intensity maintains a high recognition accuracy of over 97% under different lighting conditions compared to the fixed threshold scheme, while keeping the false positive and false negative rates at a low level (approximately 2%). This demonstrates that the strategy can effectively overcome the influence of variable light conditions in the field, ensuring that the inter-frame difference step can stably and accurately separate the seedling targets, providing high-quality and reliable input data for subsequent contour extraction, center calculation, and anomaly warning.

[0025] S4: Based on the difference image, determine whether there are seedlings in the transplanting actuator corresponding to each work state pair; if there are seedlings, extract the seedling image area in each work state pair, calculate its geometric center position, preset the seedling placement center position according to the transplanting actuator parameters, calculate the spatial offset between the seedling geometric center and the seedling placement center, and form an offset sequence related to the running phase signal. The specific method for determining the presence of seedlings based on the difference image is as follows: The difference image is binarized. Foreground pixels with a grayscale difference greater than a preset grayscale threshold are set to 255, while background pixels with a grayscale difference less than or equal to the preset grayscale threshold are set to 0, achieving clear separation between the foreground and background. Subsequently, the total number of foreground pixels in the binarized difference image is counted. The preset pixel threshold is set based on the minimum projected area of ​​the pineapple seedling. If the total number of foreground pixels is greater than the preset pixel threshold, it is determined that there are seedlings in the transplanting mechanism; otherwise, it is determined that there are no seedlings and marked as a seedling absence anomaly. The phase signal and corresponding image frame of this operation status pair are recorded simultaneously. In this scheme, the difference image is a single-channel grayscale image with a grayscale value range of 0-255, where 0 corresponds to pure black and 255 corresponds to pure white. Setting foreground pixels with a grayscale difference greater than the preset grayscale threshold to 255 marks the foreground area where seedlings may exist as pure white, while simultaneously setting subsequent background pixels to 0, i.e., marking them as pure black, achieves extreme contrast between the foreground and background, facilitating accurate extraction of the seedling outline later. When seedlings are detected, a contour extraction algorithm is used to extract the complete contour of the seedling image region based on the binarized difference image. Scattered pixels at the contour edges are removed to ensure contour accuracy. Then, based on the coordinate information of all pixels on the contour, the geometric center position of the seedling image region is calculated using the centroid calculation formula. Based on the center coordinates of the seedling delivery channel of the transplanting actuator, the coordinate parameters of the seedling delivery center position are preset. The spatial offset between the geometric center of the seedling and the delivery center is calculated using a spatial coordinate calculation method. The spatial offsets corresponding to each operation state are arranged sequentially according to the time sequence of the running phase signals to form an offset sequence that corresponds one-to-one with the running phase signals.

[0026] The formula for calculating the center of gravity is as follows: in, Here, represents the geometric center coordinates of the seedling outline, and N represents the total number of coordinates of all pixels extracted from the seedling outline. Let be the coordinates of the i-th pixel on the seedling outline; The preset seedling placement center position is the position where the geometric center of the seedling should reach when it is correctly held and without any deviation. This position, in the image coordinate system, is a fixed pixel coordinate, representing the center coordinate of the transplanting actuator. It is obtained during the system installation and debugging phase through camera calibration and mechanical alignment. This allows the transplanter to acquire multiple images within the seedling placement phase interval under no-load standard operating conditions. Image recognition is then used to determine the center point of the seedling placement channel. It is pre-stored in the system as a parameter.

[0027] S5: Set a preset offset threshold, extract values ​​in the offset sequence whose offset is greater than the offset threshold, and mark them as potential abnormal states; set a preset sliding window with a duration adapted to a complete operating cycle of the transplanting actuator, and count the proportion of potential abnormal states in each sliding window; if the proportion of potential abnormal states in the sliding window exceeds a preset proportion threshold, an abnormal warning signal is output.

[0028] Traverse the offset sequence, extract values ​​whose offset is greater than the offset threshold, and mark them as potential abnormal states; then set a sliding window of preset duration, the preset duration of which is adapted to 1-2 complete operating cycles of the transplanting actuator to ensure that it can cover one complete seedling placement process of the actuator; when the proportion of potential abnormal states in the sliding window exceeds the preset proportion threshold, output an abnormal warning signal. When an abnormal warning signal is output, simultaneously record the corresponding operating phase, offset value, and corresponding image frame of the abnormality, store the abnormal information in the local database, and send it to the transplanter control system to control the transplanter to reduce the operating speed or suspend the operation.

[0029] The offset threshold is a quantitative boundary for determining whether a seedling position in a single "operating state pair" (under a specific phase) has a "potential anomaly." If the offset exceeds this threshold, the state is marked as a "potentially abnormal state." It is determined statistically based on baseline data collected by the transplanter under normal and ideal operating conditions. Statistical analysis of the large amount of normal offset data collected can eliminate maxima caused by transient interference, and its distribution can be calculated. Preferably, the threshold can be determined using the method of "mean (μ) + 2 × standard deviation (σ)". This is a commonly used statistical threshold setting method that conforms to a normal distribution. For data conforming to a normal distribution, the probability that the data point falls within the range of μ ± 2σ is 95.4%. The core purpose of setting up the sliding window is to avoid misjudgment caused by a single potential abnormal state, ensure the accuracy and reliability of abnormal warnings, and adapt to the actual scenario of continuous operation of pineapple seedling transplanters. The preset duration setting of the sliding window needs to ensure that the sliding window can completely cover a complete seedling delivery process of the transplanting actuator, ensuring the comprehensiveness and rationality of abnormal statistics. A complete operating cycle of the transplanting actuator corresponds to a complete seedling transplanting process. The duration of the sliding window is adapted to this cycle, which can ensure that the statistical data in each sliding window can reflect the abnormal situation in a complete seedling delivery process of the actuator. A sliding window is a statistical unit of time or data points used to slide across a continuous offset sequence, analyzing the proportion of "potentially abnormal states" within the window to filter out transient interference and detect persistent faults; the specific setup method is as follows: Determine the complete operating cycle The time required for the transplanting actuator to complete one full cycle of "seedling picking-transporting-seedling placement-resetting". This can be obtained directly from the design parameters of the transplanter.

[0030] Set window duration T: T = n × Where n is typically 1 or 2, ensuring coverage of a complete seedling delivery process for the executing agency while maintaining overall efficiency. For example, if... If the time is 1 second, then the sliding window duration can be set to 2 seconds (n=1).

[0031] Converted to number of data points: Since the system processes data in frames, the final window size is reflected in the number of "job status pairs" it contains. Window size (N) = T × acquisition frame rate (fps). For example, if T = 2 seconds and the acquisition frame rate fps = 20Hz, then N = 2 × 20 = 40, meaning that a sliding window contains 40 consecutive offset data points.

[0032] The aforementioned percentage threshold is the critical proportion used to determine whether the concentration of "potential abnormal states" within the sliding window is sufficient to trigger a system warning. Considering the precision requirements of pineapple seedling transplanting operations, the operating characteristics of the transplanting actuator, and environmental interference from open-air operations, setting it too low may lead to false alarms due to momentary interference from a few frames of images, affecting operational efficiency. Setting it too high may prevent timely warnings of genuine, persistent faults, resulting in missed alarms. This application sets the threshold based on fault simulation testing. In the test, multiple sets of typical faults of different durations and degrees (such as jamming and folding) are artificially introduced to detect changes in the percentage of abnormalities within the sliding window. Multiple comparative experiments have verified that when the percentage of potential abnormal states within the sliding window exceeds 30%-50% of the total in the offset sequence, it can balance the accuracy of fault detection with the timeliness of warnings. In specific implementation, it can be initially set to 40% and then fine-tuned based on actual field test results.

[0033] In this embodiment, the abnormal warning is not directly triggered based on the offset result of a single frame image. Instead, frames with offsets exceeding the offset threshold are first marked as potential abnormal states. Then, within a sliding window that matches the operating cycle of the transplanting actuator, the proportion of potential abnormal states appearing in the window is counted, and this proportion is compared with a preset proportion threshold to determine whether to output an abnormal warning signal.

[0034] The reason for using the sliding window ratio is that factors such as sunlight reflection, mud and water blockage, mechanical vibration, motion blur, and differential residue in open field operations may cause a small number of frames to have offset anomalies. These anomalies are usually short-lived and sporadic, and belong to instantaneous interference. However, real faults such as jamming, clamping offset, seedling snagging, and folding often occur continuously within an operating cycle, making the offset anomalies appear continuously or in patches in time. Therefore, by using the potential anomaly ratio within the sliding window, instantaneous interference and continuous faults can be effectively distinguished, reducing false alarms and ensuring timely early warning.

[0035] To ensure that the statistical caliber is consistent with the work cycle, this embodiment first determines the time required for the transplanting actuator to complete one complete operation cycle of seedling picking, transporting, placing, and resetting. Based on this operation cycle, the duration of the sliding window is set. Preferably, the sliding window covers one complete operation cycle. In operation sites with greater vibration of the actuator or more drastic changes in light, the duration of the sliding window can also be set to two complete operation cycles to enhance statistical stability.

[0036] Subsequently, the duration of the sliding window is converted into the number of frames within the window based on the camera's acquisition frame rate, so that each sliding window contains a continuous pair of operation states that correspond one-to-one with the phase signal. This ensures that the key phase segments involved in a complete seedling placement process are covered within the same statistical window, providing a consistent data basis for the subsequent calibration and use of the percentage threshold.

[0037] In this embodiment, the threshold for the proportion of potential abnormal states within the sliding window is set to 30% to 50%. This is mainly because under normal operating conditions, due to factors such as strong light reflection, blade swaying, lens contamination, and slight blurring, a small number of frames may be mislabeled as potential abnormalities. However, such mislabeling usually only accounts for a small proportion within the sliding window. If the proportion threshold is set too low, for example, below about 30%, the aforementioned instantaneous interference can reach the trigger point within the statistical window, leading to frequent warnings, reducing the continuity of operation, and increasing the false alarm rate. Therefore, the proportion threshold needs to be higher than the upper limit that "occasional potential abnormalities may reach" in normal operation in order to reflect the anti-interference significance of the sliding window.

[0038] The threshold should not be too high. Real faults often cause multiple consecutive frames of abnormal offset within the window. However, in the early or moderate stages of a fault, the abnormal frames may be mainly concentrated in key phase segments such as seedling picking, transportation, or seedling placement, rather than covering all phases within the window. Therefore, their proportion may be moderate. If the threshold is set too high, for example, above about 50%, the system often needs to wait until the fault develops into a more serious state before triggering an early warning, which can easily lead to missed reports or delayed warnings and fail to meet the real-time monitoring requirements for transplanting quality. Therefore, controlling the upper limit of the threshold to about 50% is more conducive to timely early warning of moderate and continuous faults.

[0039] Based on the above two reasons, limiting the percentage threshold to the range of 30% to 50% can suppress false alarms caused by transient interference while maintaining good sensitivity to persistent faults, thus balancing the accuracy of anomaly detection with the timeliness of early warning.

[0040] To avoid providing only multiple options without a deterministic operation, this embodiment uses the following fixed procedure to calibrate the percentage threshold, ultimately obtaining a specific threshold value falling within the range of 30% to 50%. The calibration procedure is as follows: Under the condition that the transplanter is confirmed to be fault-free and the seedling supply is normal, the operation data is collected in representative lighting environments such as typical sunny strong light and cloudy weak light, and the proportion of potential anomalies in multiple sliding windows is recorded. The purpose of this step is to obtain the fluctuation range of the proportion under normal working conditions and to evaluate the statistical limit level of instantaneous interference.

[0041] Statistical analysis is performed on the percentage data collected under normal operating conditions to obtain its average level and fluctuation dispersion. A certain safety margin is added to the average level as the upper limit of the percentage that may be reached under normal operating conditions. This upper limit is used to ensure that even if short-term reflection or vibration occurs during normal operation, it is not easy to trigger the early warning, thereby controlling false alarms.

[0042] By manually introducing or reproducing typical faults, such as clamping offset, conveying jam, seedling hanging, and seedling inlet blockage, data is collected at the same frame rate and the same sliding window length. The proportion of potential anomalies in multiple sliding windows is statistically analyzed. The purpose of this step is to obtain the lower limit of the proportion under fault conditions, that is, what proportion of abnormal frames a real fault would usually cause.

[0043] Statistical analysis is performed on the proportion of fault conditions to obtain its average level and dispersion. A certain margin is reserved below the average level as the lower bound of the fault proportion. This lower bound is used to ensure that as long as a persistent fault occurs, the system can exceed the threshold with a high probability at the window statistical level, avoiding missed or delayed reporting.

[0044] A compromise value is taken between the upper limit of normal operating conditions and the lower limit of fault operating conditions as the percentage threshold, which is significantly higher than the normal fluctuation level and significantly lower than the typical fault level. If the compromise calculation result deviates from the applicable range of the project, it is limited to the range of 30% to 50% to ensure that the threshold has both anti-interference ability and early warning sensitivity.

[0045] When the equipment is deployed for the first time or the above calibration data collection has not been completed on site, this embodiment sets the initial percentage threshold to 40%. This value is in the middle of the range of 30% to 50%, which can provide a relatively robust initial balance under uncalibrated conditions. On the one hand, it can effectively filter false alarms caused by sporadic instantaneous interference frames. On the other hand, it can quickly trigger an early warning when a continuous fault occurs and forms a series of abnormal frames. After accumulating enough normal and fault window samples, the 40% will be calibrated and updated according to the above calibration process.

[0046] like Figure 2 As shown, this application also provides an image analysis-based pineapple transplanter operation monitoring system for performing the above-mentioned monitoring method, including: Benchmark library construction module: Constructs a spatial feature benchmark library containing transplanted actuators. The spatial feature benchmark library presets no-load standard images of the actuators under each operating phase and sets the corresponding operating phase intervals. Data synchronization and association module: receives dynamic image streams of transplanting actuators in real time during transplanting operations, and synchronously acquires the operating phase signals of transplanting actuators corresponding to the dynamic image streams. It then performs time-series logical association between each frame image sequence in the dynamic image stream and the operating phase signals to form operation status pairs corresponding to each operating phase. Image difference processing module: By matching the running phase signal in each job state pair with the running phase interval in the spatial feature reference library, the corresponding no-load standard image is determined. The image in each job state pair is preprocessed and background difference processing is performed with the corresponding no-load standard image to obtain the difference image. Seedling offset detection module: Based on the difference image, it determines whether there are seedlings in the transplanting actuator corresponding to each operation state pair; if there are seedlings, it extracts the seedling image area in each operation state pair, calculates its geometric center position, presets the seedling placement center position according to the transplanting actuator parameters, calculates the spatial offset between the seedling geometric center and the seedling placement center, and forms an offset sequence related to the running phase signal; Anomaly warning and judgment module: a preset offset threshold is set, and values ​​with offset values ​​greater than the offset threshold in the offset sequence are extracted and marked as potential abnormal states; a preset sliding window with a duration adapted to a complete operating cycle of the transfer actuator is used to count the proportion of potential abnormal states in each sliding window; if the proportion of potential abnormal states in the sliding window exceeds a preset proportion threshold, an anomaly warning signal is output.

[0047] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0048] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0049] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0050] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for monitoring pineapple transplanter operations based on image analysis, characterized in that, The specific steps include: S1: Construct a spatial feature benchmark library containing transplanting actuators. The spatial feature benchmark library presets no-load standard images of each operating phase of the actuator and sets the corresponding operating phase intervals. S2: Receive the dynamic image stream of the transplanting actuator during the transplanting operation in real time, and synchronously acquire the operating phase signal of the transplanting actuator corresponding to the dynamic image stream. Then, perform time-series logical association between each frame image sequence in the dynamic image stream and the operating phase signal to form a work state pair corresponding to each operating phase. S3: By matching the running phase signal in each job state pair with the running phase interval in the spatial feature reference library, determine the corresponding no-load standard image, preprocess the image in each job state pair, and perform background difference processing with the corresponding no-load standard image to obtain the difference image. S4: Based on the difference image, determine whether there are seedlings in the transplanting actuator corresponding to each work state pair; if there are seedlings, extract the seedling image area in each work state pair, calculate its geometric center position, preset the seedling placement center position according to the transplanting actuator parameters, calculate the spatial offset between the seedling geometric center and the seedling placement center, and form an offset sequence related to the running phase signal. S5: Set a preset offset threshold, extract values ​​in the offset sequence whose offset is greater than the offset threshold, and mark them as potential abnormal states; set a preset sliding window with a duration adapted to a complete operating cycle of the transplanting actuator, and count the proportion of potential abnormal states in each sliding window; if the proportion of potential abnormal states in the sliding window exceeds a preset proportion threshold, an abnormal warning signal is output.

2. The method for monitoring pineapple transplanter operations based on image analysis according to claim 1, characterized in that: The operating phase interval is a continuous range of phase values, and the operating phase intervals do not overlap.

3. The method for monitoring pineapple transplanter operations based on image analysis according to claim 2, characterized in that: When constructing the spatial feature reference library, the transplanting actuator needs to be operated under no-load conditions in all phases. Images under each operating phase are collected and then processed by denoising and grayscale to serve as no-load standard images. The preprocessing is consistent with the image preprocessing steps in step S3 for the working state, ensuring the uniformity of the grayscale reference.

4. The method for monitoring pineapple transplanter operations based on image analysis according to claim 1, characterized in that: The matching of the operating phase signal and the operating phase interval adopts the interval inclusion determination method: the specific value of the operating phase signal in the operation status is compared with the boundary value of each operating phase interval in the spatial feature reference library. If the value falls within the range of a certain operating phase interval, the no-load standard image corresponding to that operating phase interval is called as the background reference image.

5. The image analysis-based monitoring method for pineapple transplanter operations according to claim 4, characterized in that: The steps for preprocessing the image in each job state pair include denoising, grayscale conversion, image enhancement, and edge smoothing in sequence. The background differentiation process adopts the inter-frame difference method, which calls the unloaded standard image that is phase-matched with the current job state pair and has undergone the same preprocessing. The grayscale difference between the current preprocessed image and the unloaded standard image is calculated pixel by pixel. Pixels with grayscale difference values ​​greater than a preset grayscale threshold are marked as foreground pixels. The difference image is composed of all foreground pixels. The preprocessing process is synchronously associated with the running phase signal in S2.

6. The method for monitoring pineapple transplanter operations based on image analysis according to claim 1, characterized in that, The specific method for determining whether there are seedlings based on the difference image is as follows: The difference image is binarized, and the foreground pixels with a grayscale difference greater than a preset grayscale threshold are set to 255, and the background pixels with a grayscale difference less than or equal to the preset grayscale threshold are set to 0. Then, the total number of foreground pixels in the binarized difference image is counted. The preset pixel threshold is set according to the minimum projected area of ​​the pineapple seedling. If the total number of foreground pixels is greater than the preset pixel threshold, it is determined that there are seedlings in the transplanting actuator; otherwise, it is determined that there are no seedlings and marked as a seedling absence anomaly. The phase signal and corresponding image frame of this operation status pair are recorded simultaneously.

7. The image analysis-based monitoring method for pineapple transplanter operations according to claim 6, characterized in that: When seedlings are detected, a contour extraction algorithm is used to extract the complete contour of the seedling image region based on the binarized difference image. Scattered pixels at the contour edges are removed to ensure contour accuracy. Then, based on the coordinate information of all pixels on the contour, the geometric center position of the seedling image region is calculated using the centroid calculation formula. Based on the center coordinates of the seedling delivery channel of the transplanting actuator, the coordinate parameters of the seedling delivery center position are preset. The spatial offset between the geometric center of the seedling and the delivery center is calculated using a spatial coordinate calculation method. The spatial offsets corresponding to each operation state are arranged sequentially according to the time sequence of the running phase signals to form an offset sequence that corresponds one-to-one with the running phase signals.

8. The method for monitoring pineapple transplanter operations based on image analysis according to claim 7, characterized in that: Traverse the offset sequence, extract values ​​whose offset is greater than the offset threshold, and mark them as potential abnormal states; then set a sliding window of preset duration, the preset duration of which is adapted to 1-2 complete operating cycles of the transplanting actuator, to ensure that it can cover one complete seedling placement process of the actuator.

9. A method for monitoring pineapple transplanter operations based on image analysis according to claim 8, characterized in that: When an abnormal warning signal is output, the corresponding running phase, offset value and corresponding image frame are recorded synchronously, and the abnormal information is stored in the local database.

10. A pineapple transplanter operation monitoring system based on image analysis, used to execute the monitoring method of any one of claims 1-9, characterized in that, include: Benchmark library construction module: Constructs a spatial feature benchmark library containing transplanted actuators. The spatial feature benchmark library presets no-load standard images of the actuators under each operating phase and sets the corresponding operating phase intervals. Data synchronization and association module: Receives dynamic image streams from the transplanting actuator during the transplanting operation in real time, and synchronously acquires the operating phase signals of the transplanting actuator corresponding to the dynamic image stream. It then performs time-series logical association between each frame of the image stream and the operating phase signals to form operation status pairs corresponding to each operating phase. Image difference processing module: By matching the running phase signal in each job state pair with the running phase interval in the spatial feature reference library, the corresponding no-load standard image is determined. The image in each job state pair is preprocessed and background difference processing is performed with the corresponding no-load standard image to obtain the difference image. Seedling offset detection module: Based on the difference image, it determines whether there are seedlings in the transplanting actuator corresponding to each operation state pair; if there are seedlings, it extracts the seedling image area in each operation state pair, calculates its geometric center position, presets the seedling placement center position according to the transplanting actuator parameters, calculates the spatial offset between the seedling geometric center and the seedling placement center, and forms an offset sequence related to the running phase signal; Anomaly warning and judgment module: preset offset threshold, extract values ​​in the offset sequence whose offset is greater than the offset threshold, and mark them as potential abnormal states; A sliding window with a preset duration that matches a complete operating cycle of the transplanting actuator is used to count the percentage of potential abnormal states within each sliding window. If the percentage of potential abnormal states within a sliding window exceeds a preset percentage threshold, an abnormality warning signal is output.

Images