Machine vision energized straw picking state real-time identification method for self-propelled bundling machine

Abstract

The invention relates to the technical field of agricultural machinery intelligent control, in particular to a machine vision enabling self-propelled bundling machine straw picking state real-time identification method, which comprises the following steps of: acquiring a continuous multispectral image covering an operation area and preprocessing the continuous multispectral image to generate a standardized image sequence; performing straw region segmentation and spectral unmixing on the image sequence, extracting pure straw spectral features, calculating real-time moisture content, and establishing a dynamic density model in combination with visual volume to form straw physical attribute features; calculating straw dynamic behavior characteristics based on the straw skeleton, the local topological characteristics and the high-speed image optical flow; the physical attribute features and the dynamic behavior features are input into a state recognition model, a straw picking state sequence and a winding risk sequence are generated, a specific intervention action is triggered when the risk exceeds a threshold value, and intelligent control over straw picking state real-time monitoring and blocking early warning is achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for agricultural machinery, and in particular to a machine vision-enabled method for real-time identification of straw picking status in a self-propelled baler. Background Technology

[0002] Agricultural machinery straw collection systems typically rely on single-spectrum or RGB visual information for straw volume measurement and feeding control, which cannot accurately perceive the physical properties of straw. The system estimates the feeding amount by the spatial area occupied by straw in the image, but ignores the influence of straw moisture content, density, and background interference on weight, resulting in a single perception dimension and control strategies that can only be adjusted based on volume information. Some studies have attempted to use traditional optical or near-infrared methods to assess straw moisture, but these methods suffer from problems such as inaccurate band selection, severe spectral information mixing, and poor real-time performance.

[0003] In existing technologies, straw vision systems cannot distinguish between the physical density and moisture content of straw, resulting in the inability to differentiate between wet and dry straw, leading to inaccurate estimation of feed amount and inaccurate control of straw feeding. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a machine vision-enabled real-time identification method for straw picking status of a self-propelled baler, aiming to improve the problem of inaccurate feeding estimation and resulting straw feeding control failure.

[0005] In a first aspect, the present invention provides the following technical solution: a machine vision-enabled method for real-time identification of straw picking status in a self-propelled baler, comprising the following steps: A continuous multispectral image sequence of the straw-covered operation area was acquired, and the image sequence was preprocessed to generate a standardized image sequence. Straw region segmentation is performed on standardized image sequences to generate straw mask sequences, and consistency processing is performed. At the same time, unmixing analysis is performed on straw spectral information to remove background interference and obtain pure straw spectral features. Real-time moisture content is calculated based on the pure straw spectrum, and a dynamic density model is established by combining visual volume to generate the physical property characteristics of straw. Skeleton features are extracted, local structure is analyzed, and topological features are quantized from the straw mask sequence. Optical flow is calculated by combining high-speed image sequences to obtain the straw motion velocity vector and orientation field, thereby generating dynamic behavior features of the straw. The physical properties and dynamic behavior characteristics of straw are input into the state recognition model to determine the straw picking state of each frame. At the same time, the entanglement risk coefficient is calculated by combining the entanglement topology index and long linear density to generate state sequence and risk sequence. The status sequence and risk sequence are organized and continuously processed to form real-time straw picking status information, and specific intervention action sequence is triggered when the entanglement risk exceeds the set threshold.

[0006] By adopting the above technical solution, real-time and accurate identification of the straw picking status during the operation of the self-propelled baler is achieved; it can distinguish straw with different physical densities and moisture contents, avoid treating wet straw and dry straw the same, thereby improving the accurate estimation of the feeding amount and solving the problem of inaccurate straw feeding control in the existing technology.

[0007] Preferably, the preprocessing of the image sequence includes: Noise suppression for continuous multispectral image sequences; Perform brightness equalization processing on the noise-suppressed image; Perform geometric distortion correction on brightness-equalized images; The corrected image is cropped to extract the effective image covering the work area; The cropped images are then aligned between frames to generate a standardized image sequence.

[0008] Preferably, the straw area segmentation includes: The preprocessed standardized image sequence is input into the image segmentation model to identify the straw region in the image; The initial straw mask output by the segmentation model is binarized to distinguish straw pixels from non-straw pixels; Morphological processing of the binarized mask is performed, including erosion, dilation, and opening / closing operations; The processed mask is subjected to cross-frame consistency processing. Through inter-frame registration, position alignment and pixel-level fusion, a continuous frame straw mask sequence is generated. Noise removal and small-area filtering are performed on continuous frame straw mask sequences.

[0009] Preferably, the unmixing analysis of the straw spectral information includes: Extract the multispectral pixel values ​​corresponding to the straw region from consecutive frame normalized images to form the original straw spectral data; Endmember selection was performed on the raw straw spectral data to determine pure straw endmembers and existing background endmembers. Using a spectral unmixing algorithm, the mixed spectrum of each pixel is decomposed into the contribution ratio of different endmembers, background interference is removed, and pure straw spectral features are obtained. The pure straw spectra of consecutive frames are organized into a time series to generate a spectral feature sequence, and abnormal or noisy pixels are removed or smoothed during the unmixing process.

[0010] Preferably, establishing the dynamic density model includes: The visual volume of straw is calculated based on a continuous frame sequence of straw masks, and the number of mask pixels and imaging parameters are converted into spatial volume information. The spectral features of pure straw are used to calculate the straw moisture content index for each frame, and the real-time moisture content value is obtained through a pre-calibrated spectral-moisture content mapping model. Input the visual volume of straw and the real-time moisture content into the dynamic density model, and establish the relationship between the physical properties of straw according to the set function; The straw density information of consecutive frames is organized into a time series to generate a straw physical property feature sequence; Detect and correct outliers in the model input, including filtering out or smoothing extreme moisture content or abnormal volume values.

[0011] Preferably, the skeleton feature extraction, local structure analysis, and topological feature quantization include: Skeletonization processing is performed on the continuous frame straw mask sequence to extract the linear skeleton structure of the straw; The length, orientation, and connectivity information of each straw are calculated based on the skeleton structure to form a skeleton feature set; Local structural analysis is performed on the skeleton features, including skeleton branches, intersections, and density statistics, to generate local topological features; Local topological features are globally integrated to quantify the topological indices of the overall spatial distribution of straw, including long linear density and entanglement relationship indices; The skeleton features and topological features of consecutive frames are serialized to generate a sequence of dynamic behavior features of straw. Detection and correction of abnormal skeleton points or topological anomalies, including isolated skeleton removal, connection missing completion, and sequence smoothing.

[0012] Preferably, the calculation of optical flow by combining high-speed image sequences includes: Time registration is performed on consecutive frame-normalized image sequences and corresponding straw mask sequences to ensure that the same straw target is in the same position between frames; The motion vector of each straw pixel in each frame is calculated using an optical flow algorithm, including the velocity magnitude and direction of motion. Spatial aggregation processing is performed on the optical flow vector to synthesize the motion information of neighboring pixels into the average motion vector of straw particles or skeleton units; The continuous frame optical flow data is processed into a time series to generate a sequence of dynamic behavior characteristics of straw, including velocity change trends and directional consistency information.

[0013] Preferably, the determination of the straw collection status for each frame includes: The physical property feature sequence and dynamic behavior feature sequence of straw are input into the state recognition model, which is then processed using a multi-feature fusion algorithm. Classify or regress the straw feature vector corresponding to each frame of the image to determine the straw picking status category and generate a status label for each frame. Each frame's state identifier is organized with skeleton features and local topological features into a state feature vector.

[0014] Preferably, the calculation of the entanglement risk factor includes: The state feature vector, long linear density, and real-time moisture content value of each frame are integrated to form an entanglement risk feature vector. Input the entanglement risk feature vector into a lightweight classification or regression model to calculate the entanglement risk coefficient of straw in each frame; The risk coefficients of consecutive frames are organized and smoothed over time to form a risk sequence; Frames in the risk sequence that exceed a preset threshold are marked as high-risk and a high-risk frame index is generated.

[0015] Preferably, the triggering specific intervention action sequence includes: Receive the entanglement risk sequence calculated from consecutive frames, and mark frames whose risk coefficients exceed a preset threshold; Based on the temporal order and spatial location of the marked frames, a sequence of intervention action trigger signals is generated; Action trigger signal sequence controls the baler to execute specific action combinations; Real-time monitoring of the execution status of intervention actions, and collection of straw image sequences and skeleton topological features after the action is executed; The straw topological features of the action execution results are compared with the high-risk frame index to confirm the completion status of the action sequence and record the action execution log.

[0016] The present invention has the following beneficial effects: 1. In this invention, by acquiring multispectral images that are sensitive to moisture and vegetation and performing spectral demixing, the spectral features of pure straw are extracted and the background is stripped off; by combining visual volume to establish a dynamic density model, the real-time moisture content and physical density of straw are perceived, thereby solving the problem that the existing vision system can only judge volume, cannot distinguish between wet weight and dry loose straw, and leads to inaccurate feeding control.

[0017] 2. In this invention, the consistency of the movement speed and direction of straw particles is quantified by calculating the optical flow of straw from high-speed image sequences; an early warning model is constructed by combining flowability indicators and feeding signals to identify soft blockages where the feeding is normal but the flowability is reduced, thereby solving the problem that existing static prospective analysis cannot predict the dynamic behavior of straw and the precursors of soft blockages.

[0018] 3. In this invention, the morphological features of long stalks are identified through skeleton extraction and linear structure enhancement. An entanglement topology index is established based on the skeleton topology and long linear density, and the entanglement risk is quantified by combining the moisture content. A special anti-entanglement action sequence is triggered to actively intervene in straw entanglement, thereby solving the problem that existing technologies cannot identify or prevent straw entanglement. Attached Figure Description

[0019] Figure 1 The flowchart illustrates the real-time identification method for straw picking status of a self-propelled baler powered by machine vision proposed in this invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1:

[0022] In the first embodiment of the present invention, the present invention provides a machine vision-enabled method for real-time identification of the straw picking status of a self-propelled baler, such as... Figure 1 As shown, it includes the following steps: A continuous multispectral image sequence of the straw-covered operation area was acquired, and the image sequence was preprocessed to generate a standardized image sequence. Further preprocessing of the image sequence includes: Noise suppression for continuous multispectral image sequences; Perform brightness equalization processing on the noise-suppressed image; Perform geometric distortion correction on brightness-equalized images; The corrected image is cropped to extract the effective image covering the work area; The cropped images are then aligned between frames to generate a standardized image sequence.

[0023] Specifically, the self-propelled baler acquires a continuous multispectral image sequence covering the straw processing area using a multispectral camera mounted on the machine. The image sequence contains information in the red, green, blue, and near-infrared bands. The camera frame rate f is coordinated with the machine's movement speed v to ensure continuous frame coverage of the processing area and to meet the displacement conditions between adjacent frames. ,in Indicates the displacement of the coverage area of ​​adjacent frames. To determine the maximum permissible displacement and ensure the continuity of subsequent image alignment and processing; the acquired raw image sequence contains sensor noise and environmental interference, requiring noise suppression processing, typically using Gaussian filtering or median filtering, with the following expression:

[0024] in Represents the pixel values ​​of the original image. For band indexing, is the filter weight coefficient, k is the filter window radius, and the image sequence after noise suppression provides the basic data for subsequent processing; In image brightness equalization processing, pixel brightness L is adjusted using linear or histogram equalization methods to maintain consistency throughout the image sequence. The processing formula can be expressed as:

[0025] in and These are the minimum and maximum brightness values ​​of the original image, respectively. and To equalize the target brightness range; after brightness equalization, to eliminate geometric distortion caused by camera lens distortion and viewpoint projection, geometric distortion correction of the image is required. This is achieved by using the camera intrinsic parameter matrix K and the distortion parameter D to adjust the distortion coordinates. Mapped to corrected coordinates This step ensures accurate spatial structure of the image, facilitating subsequent straw region segmentation and feature extraction; During the region cropping process, based on the machine's working width and image coverage, effective images covering the working area are extracted, and edge or irrelevant background areas are removed to ensure that the processing object is only the straw working area. The cropped image sequence then undergoes inter-frame alignment processing, and the inter-frame transformation matrix is ​​calculated through key point matching or optical flow tracing. After aligning consecutive frames, a normalized image sequence is generated, which is mathematically represented as follows: ,in This is the (i+1)th frame of the image after cropping. To ensure the consistency of the spatial position of the same straw target in consecutive frames after alignment; The resulting standardized image sequence serves as input for subsequent straw region segmentation, spectral unmixing, and dynamic behavior analysis, ensuring the accuracy and temporal continuity of image processing and providing a reliable data foundation for the entire picking status recognition process.

[0026] Straw region segmentation is performed on standardized image sequences to generate straw mask sequences, and consistency processing is performed. At the same time, unmixing analysis is performed on straw spectral information to remove background interference and obtain pure straw spectral features. Furthermore, straw area segmentation includes: The preprocessed standardized image sequence is input into the image segmentation model to identify the straw region in the image; The initial straw mask output by the segmentation model is binarized to distinguish straw pixels from non-straw pixels; Morphological processing of the binarized mask is performed, including erosion, dilation, and opening / closing operations; The processed mask is subjected to cross-frame consistency processing. Through inter-frame registration, position alignment and pixel-level fusion, a continuous frame straw mask sequence is generated. Noise removal and small-area filtering are performed on continuous frame straw mask sequences.

[0027] Further, the unmixing analysis of the straw spectral information includes: Extract the multispectral pixel values ​​corresponding to the straw region from consecutive frame normalized images to form the original straw spectral data; Endmember selection was performed on the raw straw spectral data to determine pure straw endmembers and existing background endmembers. Using a spectral unmixing algorithm, the mixed spectrum of each pixel is decomposed into the contribution ratio of different endmembers, background interference is removed, and pure straw spectral features are obtained. The pure straw spectra of consecutive frames are organized into a time series to generate a spectral feature sequence, and abnormal or noisy pixels are removed or smoothed during the unmixing process.

[0028] Specifically, a standardized image sequence is input into a pre-trained image segmentation model. The model, based on a convolutional neural network or a similar deep learning structure, performs pixel-level classification on each frame of the image to identify the pixels in the straw region and the background pixels. The initial straw mask output by the model is a grayscale image or a probability image. It needs to be clearly distinguished from non-straw pixels through threshold binarization to generate a binary mask. To ensure the continuity of the straw region boundary and the structural integrity, the binary mask is subjected to morphological processing, including erosion, dilation and opening / closing operations. The erosion operation is used to remove small noise, the dilation operation is used to restore the straw edge information, and the opening / closing operation is used to optimize local connectivity and smooth the boundary. After morphological processing, the mask sequence undergoes cross-frame consistency processing. Through inter-frame registration, position alignment, and pixel-level fusion, the mask pixels corresponding to the same straw target in consecutive frames are aligned and fused to generate a continuous frame straw mask sequence. Further noise point removal and small region filtering are performed on the continuous frame mask sequence to eliminate isolated points or misidentified areas with too small an area, ensuring that the mask sequence retains only the main straw area, forming an accurate mask sequence for subsequent dynamic analysis. In the process of straw spectral unmixing analysis, multispectral pixel values ​​are extracted from the mask region of each frame of standardized image to form the original straw spectral data. ,in For pixel coordinates, Spectral band indexing; endmember selection is performed on the raw spectral data to determine pure straw endmembers. and background elements Endmember selection can be performed based on spectral feature clustering or manual calibration; using a linear spectral unmixing model, the mixed spectrum of each pixel is decomposed into different endmember contribution ratios. and The formula is:

[0029] in The unmixing residuals are represented; the spectral characteristics of pure straw are obtained by stripping away background endmember contributions. ; Organize the pure spectral features of consecutive frames into a time series to form a spectral feature sequence. Furthermore, abnormal or noisy pixels in the sequence are removed or smoothed, for example, by median filtering or moving average filtering, to achieve the continuity and stability of the spectral sequence. The final generated straw mask sequence and pure spectral feature sequence provide accurate input for subsequent calculation of straw physical properties and dynamic behavior analysis, ensuring that the data foundation of the entire state recognition process is reliable and consistent.

[0030] Real-time moisture content is calculated based on the pure straw spectrum, and a dynamic density model is established by combining visual volume to generate the physical property characteristics of straw. Furthermore, establishing a dynamic density model includes: The visual volume of straw is calculated based on a continuous frame sequence of straw masks, and the number of mask pixels and imaging parameters are converted into spatial volume information. The spectral features of pure straw are used to calculate the straw moisture content index for each frame, and the real-time moisture content value is obtained through a pre-calibrated spectral-moisture content mapping model. Input the visual volume of straw and the real-time moisture content into the dynamic density model, and establish the relationship between the physical properties of straw according to the set function; The straw density information of consecutive frames is organized into a time series to generate a straw physical property feature sequence; Detect and correct outliers in the model input, including filtering out or smoothing extreme moisture content or abnormal volume values.

[0031] Specifically, pixel counts are performed on a sequence of consecutive frames of straw mask images. The number of pixels in the straw region in each frame is then compared with camera imaging parameters, including focal length f and sensor size. and pixel resolution It is converted into spatial volume information through geometric mapping relationships. The formula is expressed as

[0032] in Let d be the set of straw pixels in the mask of frame t, and d be the imaging distance, which can be obtained through sensor calibration; the pure straw spectral characteristics of consecutive frames are used to determine the range of straw pixels. Extract spectral indices, such as near-infrared reflectance or specific band ratios. Through a pre-calibrated spectral-moisture mapping model Obtain the real-time moisture content of straw in each frame. Visual volume With real-time moisture content The common input dynamic density model is determined by setting a function. Establish the relationship between the physical properties of straw, among which Indicates the first The density or other physical properties of straw in frames can be represented by a function g, which can be implemented using a weighted linear combination or a nonlinear mapping method to reflect the combined effects of volume and moisture content. The density information of consecutive frames is organized into a time series to generate a sequence of straw physical property features. This is to support the input of subsequent dynamic behavior analysis and state recognition models; During model operation, outliers are detected and corrected, including those related to extreme moisture content. or abnormal volume The filtering or smoothing processes, such as median filtering, moving average, or threshold truncation, ensure that the generated straw physical property feature sequence is stable and continuous. Through the above processing, the physical property data of each frame of straw can be used for real-time analysis of straw status and subsequent entanglement risk calculation.

[0033] Skeleton features are extracted, local structure is analyzed, and topological features are quantized from the straw mask sequence. Optical flow is calculated by combining high-speed image sequences to obtain the straw motion velocity vector and orientation field, thereby generating dynamic behavior features of the straw. Furthermore, skeleton feature extraction, local structure analysis, and topological feature quantization include: Skeletonization processing is performed on the continuous frame straw mask sequence to extract the linear skeleton structure of the straw; The length, orientation, and connectivity information of each straw are calculated based on the skeleton structure to form a skeleton feature set; Local structural analysis is performed on the skeleton features, including skeleton branches, intersections, and density statistics, to generate local topological features; Local topological features are globally integrated to quantify the topological indices of the overall spatial distribution of straw, including long linear density and entanglement relationship indices; The skeleton features and topological features of consecutive frames are serialized to generate a sequence of dynamic behavior features of straw. Detection and correction of abnormal skeleton points or topological anomalies, including isolated skeleton removal, connection missing completion, and sequence smoothing.

[0034] Furthermore, calculating optical flow by combining high-speed image sequences includes: Time registration is performed on consecutive frame-normalized image sequences and corresponding straw mask sequences to ensure that the same straw target is in the same position between frames; The motion vector of each straw pixel in each frame is calculated using an optical flow algorithm, including the velocity magnitude and direction of motion. Spatial aggregation processing is performed on the optical flow vector to synthesize the motion information of neighboring pixels into the average motion vector of straw particles or skeleton units; The continuous frame optical flow data is processed into a time series to generate a sequence of dynamic behavior characteristics of straw, including velocity change trends and directional consistency information.

[0035] Specifically, the continuous frame straw mask sequence is processed by skeletonization, extracting the linear skeleton structure of the straw, and generating the corresponding skeleton map of the straw region in each frame mask through a thinning algorithm or skeleton extraction algorithm. ,in Represents skeleton pixels, Represents the non-skeleton region; calculates the length of each straw based on the skeleton structure. ,direction and connectivity Information, and form a skeleton feature set. ,in The t-th frame represents the number of straw stalks in the skeleton. Local structural analysis is performed on the skeleton features, including the number of skeleton branches, intersection locations, and local density statistics, generating local topological features. The local topological features are then weighted and integrated to generate a global topological index. For example, long linear density and entanglement relationship index It can be done through the formula:

[0036] in The area of ​​the working region is defined; the skeleton features and topological features of consecutive frames are processed into a time series to generate a sequence of dynamic behavior features of straw. During serialization, isolated skeleton points, missing connections, and topological anomalies are detected and corrected using neighborhood search, connection completion, and sequence smoothing methods to ensure the continuity and reliability of the skeleton feature sequence. Combined with high-speed image sequences, time registration is performed on consecutive frame-normalized images and corresponding skeleton masks to ensure the consistent position of the same straw target across frames. An optical flow algorithm is used to calculate the motion vector of each skeleton pixel in each frame. ,in Pixel displacements in the horizontal and vertical directions are respectively, which can be expressed by the formula:

[0037] in, The frame interval is used for spatial aggregation of optical flow vectors, which synthesizes the motion information of neighboring skeleton pixels into the average motion vector of the skeleton unit or straw particle. Furthermore, the continuous frame optical flow data is processed into a time sequence to generate a sequence of dynamic behavior characteristics of straw, including speed change trends, directional consistency, and local motion pattern information. The combination of the aforementioned skeleton features and optical flow information can be used for subsequent straw state recognition and entanglement risk calculation to form a complete dynamic behavioral feature input.

[0038] The physical properties and dynamic behavior characteristics of straw are input into the state recognition model to determine the straw picking state of each frame. At the same time, the entanglement risk coefficient is calculated by combining the entanglement topology index and long linear density to generate state sequence and risk sequence. Furthermore, the determination of the straw collection status for each frame includes: The physical property feature sequence and dynamic behavior feature sequence of straw are input into the state recognition model, which is then processed using a multi-feature fusion algorithm. Classify or regress the straw feature vector corresponding to each frame of the image to determine the straw picking status category and generate a status label for each frame. Each frame's state identifier is organized with skeleton features and local topological features into a state feature vector.

[0039] Furthermore, the calculation of the entanglement risk factor includes: The state feature vector, long linear density, and real-time moisture content value of each frame are integrated to form an entanglement risk feature vector. Input the entanglement risk feature vector into a lightweight classification or regression model to calculate the entanglement risk coefficient of straw in each frame; The risk coefficients of consecutive frames are organized and smoothed over time to form a risk sequence; Frames in the risk sequence that exceed a preset threshold are marked as high-risk and a high-risk frame index is generated.

[0040] Specifically, the straw physical property feature sequence and dynamic behavior feature sequence generated from the aforementioned steps will be input into the state recognition and risk assessment unit for joint processing. This unit first performs unified preprocessing on various features, including feature normalization, missing value imputation and time alignment, to form a feature vector sequence corresponding to each frame, providing a consistent data basis for subsequent judgment and risk calculation. When running the status recognition model, the feature vector of straw physical attributes for each frame is used. With dynamic behavior feature vector Perform feature-level or decision-level fusion to construct frame-level state feature vectors. ,in This indicates feature concatenation, dimensionality reduction, or weighted combination operations, where t is the frame index; to ensure consistency of model input, [the following is omitted as it is not explicitly stated in the original text]. Normalization is performed using either zero-mean unit variance transformation or max-min scaling. The normalization formula is as follows: ,in and These are the mean vector and standard deviation vector of the training dataset, respectively. The state recognition model can employ a classification or regression framework, selecting a lightweight neural network, decision tree, support vector machine, or other deployable multi-feature fusion algorithm based on implementation requirements. The model is trained using offline labeled samples and inferenced in an embedded environment. The model output is a state label for each frame. and optional confidence scores The state identifier of each frame is compared with the local topological features from the skeleton topology analysis. The shared organization is represented by a state feature vector, which is used for subsequent risk assessment and decision support. In the stage of constructing entanglement risk characteristics and calculating coefficients, the state feature vector is... Long linear density index With real-time moisture content Integrate into an entanglement risk feature vector To facilitate model processing, standardization or logarithmic transformation is performed on the numerical components; entanglement risk coefficient. It can be calculated using lightweight classification or regression models. If logistic regression is used to represent probabilistic risk, the formula can be used:

[0041] in The weight vector obtained during training, For bias, for function, The range of values ​​is This is used to represent the probability or degree of frame-level entanglement risk; if a regression model is used, It can be a real number and mapped to a risk interval as needed; to improve time stability, the original... For time-series smoothing, an exponentially weighted moving average can be used:

[0042] in For the smoothed risk value, The smoothing coefficient is the initial value. Can be set to Or the average risk of the training set; after smoothing, according to a preset threshold To assess the risk, the set of high-risk frames is defined as follows:

[0043] To avoid isolated false alarms, a continuous determination can be introduced, whereby a certain frame t satisfies... And in the window When the number of high-risk frames within a time period is not less than the threshold k, the time period is marked as a continuous high-risk event; where K and k are configurable parameters to balance sensitivity and stability. At the operational level, model training utilizes labeled historical job datasets for offline training and cross-validation. The training process includes feature selection, hyperparameter tuning, and model compression to facilitate deployment on edge devices. During online runtime, input features are normalized online and inferred in real time. Interpolation, pruning, or backoff strategies are implemented for missing or anomalous inputs. Anomaly detection can be based on thresholds, outlier detection, or model confidence. Make a judgment and trigger the data correction process; risk sequences and status sequences are stored by frame index for retrospective analysis and strategy decision-making. In the application and decision-making stages, high-risk frame indexes are output to the execution unit or upper-level control strategy module. The duration and spatial distribution information of high-risk events are combined to determine whether to trigger intervention actions or record suspicious operations. All judgment records and risk trajectories are logged and incorporated into subsequent model retraining and threshold recalibration processes to ensure long-term maintainability and data-driven continuous improvement.

[0044] The status sequence and risk sequence are organized and continuously processed to form real-time straw picking status information, and specific intervention action sequence is triggered when the entanglement risk exceeds the set threshold.

[0045] Furthermore, the sequences of actions that trigger specific interventions include: Receive the entanglement risk sequence calculated from consecutive frames, and mark frames whose risk coefficients exceed a preset threshold; Based on the temporal order and spatial location of the marked frames, a sequence of intervention action trigger signals is generated; Action trigger signal sequence controls the baler to execute specific action combinations; Real-time monitoring of the execution status of intervention actions, and collection of straw image sequences and skeleton topological features after the action is executed; The straw topological features of the action execution results are compared with the high-risk frame index to confirm the completion status of the action sequence and record the action execution log.

[0046] Specifically, the control unit that triggers the specific intervention sequence continuously receives smoothed risk sequences from the risk assessment module. and for those exceeding the preset threshold The frames are marked to generate a set of high-risk frames. To avoid isolated false alarms, continuous detection is further performed on H. When the number of high-risk frames within the sliding window length K is not less than the threshold k, these frames are merged into a single high-risk event interval. Each high-risk event interval is further mapped to frame-level pixel coordinates using a pixel-to-body reference coordinate system mapping function. Convert to spatial coordinates The mapping can be represented using a pinhole camera model as follows:

[0047] in For the camera Equivalent focal length in direction, Z is the main point coordinate, and Z is the line-of-sight distance or height estimate. If there is no direct depth sensor, Z can be estimated from the working height and the camera calibration relationship. Based on the spatial projection results of the time series, high-risk frames are clustered in space to obtain one or more high-risk areas, and the center position, coverage area and event duration of each area are calculated to provide spatial and temporal basic information for the generation of subsequent intervention actions. When generating the sequence of intervention action trigger signals, the predefined action strategy mapping function is used based on the risk intensity, duration, and spatial distribution of each high-risk event. Automatically select action combinations and execution order in the mapping function. This represents the maximum smoothed risk value within the event interval. Indicates the duration of the event. The area represents the high-risk zone; the action set is a sequence of parameterized action units, such as optional high-frequency vibration units, instantaneous micro-reversal units of spring teeth, and micro-speed follow-up control units. Each action unit includes configurable parameters such as execution duration, frequency, or angular displacement amplitude; the trigger signal sequence consists of a timestamp, target spatial position, action unit identifier, and parameters, and is sent to the actuator driver via the field control bus. The actuator driver converts the action unit type into a specific execution command to control the machine body drive or peripheral actions. During the execution of the action, a real-time monitoring process is initiated, continuously acquiring image sequences after the action is executed via cameras and recalculating the mask, skeleton topology features, and smoothing risk values. Simultaneously, the executed commands and timestamps are recorded for subsequent comparison and backtracking analysis. The evaluation of the execution results is completed by comparing key indicators before and after the event is triggered. The typical comparison logic is to determine whether the average risk value within an observation window after the event interval ends is lower than the backoff threshold. If a preset topology change pattern occurs and the clearing conditions are met, the event is marked as processed, and the processing frame range, action sequence identifier, execution parameters, and evaluation results are recorded in the execution log. If the clearing conditions are not met, the control unit decides whether to initiate the next round of actions based on the repetition limit policy. The repetition policy can limit the maximum number of repetitions. And the interval between each round, to avoid continuous impact on the mechanical structure; In terms of data management and recording, all triggering events, action sequences, and evaluation results are written to local logs and backhaul caches by frame index and timestamp. Log entries include event identifier, start and end frames, spatial center coordinates, selected action unit, execution parameters, action start and end times, action execution result judgment, and optional intervention annotations by operators. These logs are used for online strategy tuning, threshold recalibration, and offline model retraining, and support batch retrieval and replay analysis by event. To ensure operational safety and reliability, the action triggering process incorporates multiple detection and safety constraints, including checking the mechanical status before triggering, monitoring the actuator load and response during execution, and immediately stopping the action and recording the exception code when an anomaly is detected, so as to ensure the controllability and traceability of the action sequence in the actual working environment.

[0048] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A machine vision-enabled method for real-time identification of straw picking status in a self-propelled baler, characterized in that, Includes the following steps: A continuous multispectral image sequence of the straw-covered operation area was acquired, and the image sequence was preprocessed to generate a standardized image sequence. Straw region segmentation is performed on standardized image sequences to generate straw mask sequences, and consistency processing is performed. At the same time, unmixing analysis is performed on straw spectral information to remove background interference and obtain pure straw spectral features. Real-time moisture content is calculated based on the pure straw spectrum, and a dynamic density model is established by combining visual volume to generate the physical property characteristics of straw. Skeleton features are extracted, local structure is analyzed, and topological features are quantized from the straw mask sequence. Optical flow is calculated by combining high-speed image sequences to obtain the straw motion velocity vector and orientation field, thereby generating dynamic behavior features of the straw. The physical properties and dynamic behavior characteristics of straw are input into the state recognition model to determine the straw picking state of each frame. At the same time, the entanglement risk coefficient is calculated by combining the entanglement topology index and long linear density to generate state sequence and risk sequence. The status sequence and risk sequence are organized and continuously processed to form real-time straw picking status information, and specific intervention action sequence is triggered when the entanglement risk exceeds the set threshold.

2. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The preprocessing of the image sequence includes: Noise suppression for continuous multispectral image sequences; Perform brightness equalization processing on the noise-suppressed image; Perform geometric distortion correction on brightness-equalized images; The corrected image is cropped to extract the effective image covering the work area; The cropped images are then aligned between frames to generate a standardized image sequence.

3. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The straw region segmentation includes: The preprocessed standardized image sequence is input into the image segmentation model to identify the straw region in the image; The initial straw mask output by the segmentation model is binarized to distinguish straw pixels from non-straw pixels; Morphological processing of the binarized mask is performed, including erosion, dilation, and opening / closing operations; The processed mask is subjected to cross-frame consistency processing. Through inter-frame registration, position alignment and pixel-level fusion, a continuous frame straw mask sequence is generated. Noise removal and small-area filtering are performed on continuous frame straw mask sequences.

4. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The unmixing analysis of the straw spectral information includes: Extract the multispectral pixel values ​​corresponding to the straw region from consecutive frame normalized images to form the original straw spectral data; Endmember selection was performed on the raw straw spectral data to determine pure straw endmembers and existing background endmembers. Using a spectral unmixing algorithm, the mixed spectrum of each pixel is decomposed into the contribution ratio of different endmembers, background interference is removed, and pure straw spectral features are obtained. The pure straw spectra of consecutive frames are organized into a time series to generate a spectral feature sequence, and abnormal or noisy pixels are removed or smoothed during the unmixing process.

5. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The establishment of the dynamic density model includes: The visual volume of straw is calculated based on a continuous frame sequence of straw masks, and the number of mask pixels and imaging parameters are converted into spatial volume information. The spectral features of pure straw are used to calculate the straw moisture content index for each frame, and the real-time moisture content value is obtained through a pre-calibrated spectral-moisture content mapping model. Input the visual volume of straw and the real-time moisture content into the dynamic density model, and establish the relationship between the physical properties of straw according to the set function; The straw density information of consecutive frames is organized into a time series to generate a straw physical property feature sequence; Detect and correct outliers in the model input, including filtering out or smoothing extreme moisture content or abnormal volume values.

6. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The skeleton feature extraction, local structure analysis, and topological feature quantization include: Skeletonization processing is performed on the continuous frame straw mask sequence to extract the linear skeleton structure of the straw; The length, orientation, and connectivity information of each straw are calculated based on the skeleton structure to form a skeleton feature set; Local structural analysis is performed on the skeleton features, including skeleton branches, intersections, and density statistics, to generate local topological features; Local topological features are globally integrated to quantify the topological indices of the overall spatial distribution of straw, including long linear density and entanglement relationship indices; The skeleton features and topological features of consecutive frames are serialized to generate a sequence of dynamic behavior features of straw. Detection and correction of abnormal skeleton points or topological anomalies, including isolated skeleton removal, connection missing completion, and sequence smoothing.

7. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The calculation of optical flow by combining high-speed image sequences includes: Time registration is performed on consecutive frame-normalized image sequences and corresponding straw mask sequences to ensure that the same straw target is in the same position between frames; The motion vector of each straw pixel in each frame is calculated using an optical flow algorithm, including the velocity magnitude and direction of motion. Spatial aggregation processing is performed on the optical flow vector to synthesize the motion information of neighboring pixels into the average motion vector of straw particles or skeleton units; The continuous frame optical flow data is processed into a time series to generate a sequence of dynamic behavior characteristics of straw, including velocity change trends and directional consistency information.

8. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The determination of the straw collection status for each frame includes: The physical property feature sequence and dynamic behavior feature sequence of straw are input into the state recognition model, which is then processed using a multi-feature fusion algorithm. Classify or regress the straw feature vector corresponding to each frame of the image to determine the straw picking status category and generate a status label for each frame. Each frame's state identifier is organized with skeleton features and local topological features into a state feature vector.

9. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The calculation of the entanglement risk coefficient includes: The state feature vector, long linear density, and real-time moisture content value of each frame are integrated to form an entanglement risk feature vector. Input the entanglement risk feature vector into a lightweight classification or regression model to calculate the entanglement risk coefficient of straw in each frame; The risk coefficients of consecutive frames are organized and smoothed over time to form a risk sequence; Frames in the risk sequence that exceed a preset threshold are marked as high-risk and a high-risk frame index is generated.

10. The machine vision-enabled real-time straw picking status recognition method for self-propelled balers according to claim 1, characterized in that, The trigger-specific intervention sequence includes: Receive the entanglement risk sequence calculated from consecutive frames, and mark frames whose risk coefficient exceeds a preset threshold; Based on the temporal order and spatial location of the marked frames, a sequence of intervention action trigger signals is generated; Action trigger signal sequence controls the baler to execute specific action combinations; Real-time monitoring of the execution status of intervention actions, and collection of straw image sequences and skeleton topological features after the action is executed; The straw topological features of the action execution results are compared with the high-risk frame index to confirm the completion status of the action sequence and record the action execution log.

Images