In-field work video-based agricultural implement tillage depth measurement apparatus and method thereof

Abstract

The present invention relates to an apparatus and method for measuring the plowing depth of an agricultural implement based on a work video. An apparatus for measuring the plowing depth of an agricultural implement based on a work video according to one embodiment of the present invention is an apparatus for measuring the plowing depth of an agricultural implement based on a work video including an agricultural implement and a soil surface, which is acquired from a camera installed at the rear of a tractor. It may include a calibration unit that sets a plurality of plowing depth scale lines, which are correspondence relationships between pixel positions within the work video and actual plowing depth values, using a correction plate having a preset pattern formed thereon; a zero-point setting unit that detects a specific part of the agricultural implement as a reference target from a work video in which the plowing depth value at the start of the work is a preset initial value, and sets and stores the pixel position of the reference target as a zero reference line; and a calculation unit that detects the current pixel position of the reference target from a video frame of a work video acquired during the work, and calculates a plowing depth value corresponding to the current pixel position based on the zero reference line and the plowing depth scale lines.

Description

Technology Field

[0001] The present invention relates to an apparatus and method for measuring the tillage depth of an agricultural implement based on an operation video, and more specifically, to an apparatus and method for measuring the tillage depth of an agricultural implement based on an operation video that establishes a correspondence relationship between a pixel position and an actual tillage depth value using an operation video acquired from a camera installed at the rear of a tractor and calculates the tillage depth value based thereon. Background Technology

[0002] Plowing depth is a critical factor that directly affects work quality in agricultural machinery, and accurately monitoring it is essential for increasing efficiency and productivity. At the current level of technology, the primary method used to monitor plowing depth involves equipping separate sensors, such as depth sensors, displacement sensors, or contact sensors. These sensors detect changes in plowing depth in real time, helping operators understand the working conditions. In particular, the importance of such sensor technology is increasingly being highlighted alongside the advancement of precision agriculture.

[0003] However, the method of adding separate sensors entails several problems. First, this approach complicates the configuration and installation of agricultural machinery. Mounting the sensors requires additional space within the machine structure, and wiring work related to the sensors is also essential. This complexity demands additional time and costs during the design and manufacturing processes of the machinery. Furthermore, there is a possibility that the sensor's performance may degrade due to carelessness during installation or design flaws.

[0004] Furthermore, sensor failures, contamination, and wiring issues act as major factors that increase the burden of maintenance. Agricultural working environments provide conditions where sensors can easily become contaminated or damaged due to dust, mud, and moisture. These environmental factors shorten the lifespan of sensors and require frequent inspections and replacements. Additionally, wiring problems can disrupt signal transmission during operation, which can directly affect work quality. Consequently, these issues increase the operating costs of agricultural machinery and place a heavier burden on workers. The problem to be solved

[0005] The objective of the present invention is to provide a work image-based agricultural implement depth measuring device and a method thereof, which measure the depth of an agricultural implement based on a work image. means of solving the problem

[0006] In one embodiment of the present invention, a work image-based agricultural implement depth measuring device may be provided. The work image-based agricultural implement depth measuring device may include: a calibration unit that sets a plurality of depth scale lines, which are correspondence relationships between pixel positions within the work image and actual depth values, using a correction plate having a preset pattern formed thereon; a zero-point setting unit that detects a specific part of the agricultural implement as a reference target from a work image in which the depth value at the start of the work is a preset initial value, and sets and stores the pixel position of the reference target as a zero reference line; and a calculation unit that detects the current pixel position of the reference target from an image frame of the work image acquired during the work, and calculates a depth value corresponding to the current pixel position based on the zero reference line and the depth scale lines.

[0007] In one embodiment of the present invention, the calibration unit detects the pixel position of the pattern from the working image, calculates the correspondence relationship between the pixel position and the actual meridian value based on the actual dimensions of the pattern, and maps and sets a plurality of meridian scale lines defined by a preset meridian value interval according to the correspondence relationship.

[0008] In one embodiment of the present invention, the zero point setting unit can detect a specific part including at least one of a contour, a corner, and a surface boundary included in the upper structure of an agricultural machine as a reference target, and set and store the pixel position of the reference target as a zero point reference line.

[0009] In one embodiment of the present invention, when the current pixel position of a reference object is located between two adjacent metacenter scale lines, the operation unit can calculate a metacenter value corresponding to the current pixel position through linear interpolation based on the metacenter value corresponding to each of the two adjacent metacenter scale lines.

[0010] In one embodiment of the present invention, a method for measuring the tillage depth of an agricultural implement based on a work image may be provided. The method may include: a calibration step of setting a plurality of tillage scale lines, which are correspondences between pixel positions within the work image and actual tillage values, using a correction plate having a preset pattern formed thereon; a zero-point setting step of detecting a specific part of the agricultural implement as a reference target from a work image in which the tillage value at the start of the work is a preset initial value, and setting and storing the pixel position of the reference target as a zero reference line; and a calculation step of detecting the current pixel position of the reference target from an image frame of a work image acquired during the work, and calculating a tillage value corresponding to the current pixel position based on the zero reference line and the tillage scale lines.

[0011] In one embodiment of the present invention, the calibration step may detect the pixel location of the pattern from the working image, calculate the correspondence relationship between the pixel location and the actual meridian value based on the actual dimensions of the pattern, and map and set a plurality of meridian scale lines defined by a preset meridian value interval according to the correspondence relationship.

[0012] In one embodiment of the present invention, the zero point setting step may detect a specific part including at least one of a contour, a corner, and a surface boundary included in the upper structure of an agricultural machine as a reference target, and set and store the pixel position of the reference target as a zero point reference line.

[0013] In one embodiment of the present invention, the calculation step can calculate a meridian value corresponding to the current pixel position through linear interpolation based on the meridian value corresponding to each of the two adjacent meridian scale lines when the current pixel position of a reference target is located between two adjacent meridian scale lines. Effects of the invention

[0014] According to one embodiment of the present invention, a device comprising a calibration unit, a zero-point setting unit, and a calculation unit is provided to measure the tillage depth of an agricultural implement based on a work video acquired from a camera installed at the rear of a tractor. This enables tillage depth measurement without the need for additional mounting of a separate sensor, thereby simplifying configuration and installation and reducing the burden and cost of maintenance. According to one embodiment of the present invention, a correspondence relationship is established between a pixel position within the work video and an actual tillage depth value. The pixel position of a reference target at the start of the work is set as the zero reference line. By detecting the current pixel position of the reference target in each video frame during the work and calculating the tillage depth value, accurate tillage depth values ​​can be measured in real time even during the work. Furthermore, according to the present invention, continuous tillage depth values ​​can be derived through linear interpolation when located between adjacent scale lines, thereby further improving the precision of tillage depth measurement. Brief explanation of the drawing

[0015] FIG. 1 is a block diagram illustrating the configuration of a work video-based agricultural tool depth measurement system according to one embodiment of the present invention. FIG. 2 is an illustrative diagram for explaining the configuration of a tractor according to one embodiment of the present invention. FIG. 3 is an illustrative diagram for explaining a calibration unit according to an embodiment of the present invention. FIG. 4 is an illustrative diagram for explaining a zero point setting unit according to an embodiment of the present invention. FIG. 5 is an illustrative diagram for explaining a calculation unit according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating the schematic flow of a method for measuring the depth of an agricultural machine based on a work video according to one embodiment of the present invention. FIG. 7 is a flowchart illustrating the detailed flow of an operation step according to one embodiment of the present invention. Specific details for implementing the invention

[0016] The following detailed description of the invention refers to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that various embodiments of the invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the invention is limited only by the appended claims, including all equivalents to those claimed therein, provided appropriately described. Similar reference numerals in the drawings refer to the same or similar functions across various aspects.

[0017] The components according to the present invention are defined by functional distinction rather than physical distinction, and can be defined by the functions each performs. Each component may be implemented as hardware or as program code and processing units that perform each function, and the functions of two or more components may be included and implemented in a single component. Therefore, it should be noted that the names assigned to the components in the following embodiments are not intended to physically distinguish each component but are assigned to imply the representative function performed by each component, and that the technical concept of the present invention is not limited by the names of the components.

[0018] FIG. 1 is a block diagram illustrating the configuration of a work video-based agricultural tool depth measurement system (1) according to one embodiment of the present invention.

[0019] Referring to FIG. 1, the work video-based agricultural implement depth measurement system (1) may be a system for calculating the depth of an agricultural implement (220) in real time. The work video-based agricultural implement depth measurement system (1) can measure the depth to which the working part digs from the soil surface at the site. The work video-based agricultural implement depth measurement system (1) may include a work video-based agricultural implement depth measurement device (10) and a tractor (20). The work video-based agricultural implement depth measurement system (1) may not additionally attach a separate depth sensor or displacement sensor to the agricultural implement (220). The work video-based agricultural implement depth measurement system (1) can calculate the depth value using only the work video obtained from a camera installed at the rear of the tractor (20).

[0020] The tractor (20) may include a tractor body (210), an agricultural implement (220), and an image acquisition unit (230). The tractor body (210) may form the main structure of the tractor (20). The agricultural implement (220) may be coupled to the tractor body (210). The agricultural implement (220) may perform the task of cultivating the soil. The image acquisition unit (230) may perform the function of acquiring an image of the work, including the agricultural implement (220) and the soil surface.

[0021] The image acquisition unit (230) may be installed at the rear or rear upper part of the tractor (20). The image acquisition unit (230) may set the shooting field of view so that the agricultural implement (220) and the soil surface are included together within the same image frame. The work image may be a sequence of image frames continuously acquired during the work. Each image frame may be provided to the work image-based agricultural implement depth measuring device (10) in chronological order. For example, the image acquisition unit (230) may include a camera, a lens, an image sensor, an image transmission interface, a lighting device, a gyroscope sensor, and an image signal processor (ISP). However, the present invention is not limited thereto.

[0022] The work video-based agricultural implement depth measuring device (10) may include a calibration unit (110), a zero point setting unit (120), and a calculation unit (130). The work video-based agricultural implement depth measuring device (10) may be implemented as an electronic device including a processor, memory, and a communication module. The work video-based agricultural implement depth measuring device (10) may be integrally mounted on a tractor (20). The work video-based agricultural implement depth measuring device (10) may be implemented as a separate calculation unit outside the tractor (20). The work video-based agricultural implement depth measuring device (10) may be connected to an image acquisition unit (230) via a wired or wireless connection.

[0023] The calibration unit (110) may use a correction plate having a pre-set pattern formed thereon. The calibration unit (110) may establish a correspondence relationship between pixel positions within the working image and actual meridian values. The calibration unit (110) may map and store multiple meridian scale lines, defined by pre-set meridian value intervals according to the established correspondence relationship, onto the image coordinate system. Due to the camera's installation angle or shooting distance, pixel displacement in the image may not be proportional to the actual displacement. The calibration unit (110) may correct installation deviations by establishing a correspondence relationship based on the correction plate.

[0024] The zero point setting unit (120) may use a work image in which the tillage depth value at the start of the work is a preset initial value. The zero point setting unit (120) may detect a specific part of the agricultural implement (220) as a reference target from the work image. The zero point setting unit (120) may set and store the pixel location of the detected reference target as a zero point reference line. The specific part of the agricultural implement (220) may include at least one of a contour, a corner, and a surface boundary included in the upper structure of the agricultural implement (220). In one embodiment of the present invention, the preset initial value may be 0, which is the tillage depth value in a state where the agricultural implement (220) has contacted the soil surface but has not started cultivation.

[0025] The calculation unit (130) can detect the current pixel position of a reference target from an image frame of a work video acquired during the work. The calculation unit (130) can calculate a hardness value corresponding to the current pixel position based on a zero reference line and a hardness scale line. The calculation unit (130) can determine which hardness scale line the current pixel position is closest to. The calculation unit (130) can determine whether the current pixel position exists between two adjacent hardness scale lines. The calculation unit (130) can calculate a hardness value based on the determination result. The calculated hardness value can be output in various forms, such as screen display, log file storage, or linkage with a tractor controller.

[0026] In one embodiment of the present invention, the zero point setting unit (120) and the calculation unit (130) may be configured to perform additional auxiliary processing. During operation, lighting conditions may change rapidly or the image may shake due to vibration of the tractor (20). The zero point setting unit (120) and the calculation unit (130) may perform auxiliary processing to respond to such cases. For example, the auxiliary processing may include template storage, brightness and contrast normalization, frame stabilization, temporal filtering, image correction, noise removal, and feature point matching. However, the present invention is not limited thereto. The auxiliary processing may improve the continuity and reliability of the depth calculation.

[0027] FIG. 2 is an illustrative diagram for explaining the configuration of a tractor (20) according to one embodiment of the present invention.

[0028] Regarding the tractor (20), details that overlap with those described in FIG. 1 may be omitted. The tractor (20) can provide a physical environment that generates work images.

[0029] The tractor body (210) may be the main structure of the tractor (20). For example, the tractor body (210) may include a power source for driving, a transmission, wheels, a steering device, a driver's seat, a power supply, and a hitch structure for connecting implements. However, the present invention is not limited thereto. The present invention may not be limited to a specific driving method of the tractor body (210). Furthermore, the present invention may be applied to tractor structures commonly used by those skilled in the art.

[0030] The agricultural implement (220) may be a device for handling soil that is coupled to the rear of the tractor body (210). The agricultural implement (220) may perform soil tillage, leveling, or rotary work. The agricultural implement (220) may include a working part that performs work in direct contact with the soil, and a frame and upper structure that support the working part. The working part may be, for example, a blade, a rotor, a disk, a harrow, a plow, a tillage blade, a rotary blade, etc. However, the present invention is not limited thereto.

[0031] The above-mentioned upper structure may be a part that is relatively stably exposed within the camera's field of view. The zero-point setting unit (120) can detect a specific part that includes at least one of the contours, corners, and surface boundaries included in the upper structure of the agricultural implement (220) as a reference target. The zero-point setting unit (120) can set and save the pixel location of the reference target as a zero reference line from the work image in which the diagonal depth value at the start of the work is a preset initial value. Accordingly, the agricultural implement (220) can be installed so that the upper structure is included within the work image. The reference target may be selected as a different part depending on the type of agricultural implement (220).

[0032] FIG. 2 shows the actual tillage depth of the agricultural implement (220) D t It can be illustrated as. The above D tmay refer to the depth to which the working part digs from the soil surface. The above D t Depending on the type of work machine, it may be referred to as the work depth or the tillage depth. The calculation unit (130) can detect the current pixel position of the reference target from the image frame of the work image acquired during the work. The calculation unit (130) can calculate a hardness value corresponding to the current pixel position based on the zero reference line and the hardness scale line.

[0033] The image acquisition unit (230) is installed at the rear of the tractor (20) to acquire work images. The image acquisition unit (230) can acquire work images including the agricultural implement (220) and the soil surface. FIG. 2 may illustrate an example where the image acquisition unit (230) is installed at the rear upper part of the tractor body (210) so that the agricultural implement (220) and the ground are captured simultaneously. The installation position and shooting angle of the image acquisition unit (230) can be set so that the upper structure of the agricultural implement (220) and the soil surface are included together within the frame.

[0034] The image acquisition unit (230) may include a camera, lens, image sensor, lighting device, gyroscope sensor, etc. The work image may be acquired as a continuous sequence of frames. Each frame may be input to the calculation unit (130) over time so that the current pixel position of the reference target can be repeatedly calculated. Additionally, the image acquisition unit (230) may include a wired or wireless communication interface for transmitting the image signal to the work image-based agricultural tool depth measuring device (10). For example, the communication interface may be a Universal Serial Bus (USB), Ethernet, a Controller Area Network (CAN) gateway, Wi-Fi, Bluetooth, Zigbee, or a cellular communication module. However, the present invention is not limited thereto.

[0035] In one embodiment of the present invention, the image acquisition unit (230) may be configured to provide shooting parameters as additional information along with the working image frame. The shooting parameters may include at least one of an exposure time, a gain value, a white balance, and a frame timestamp. The operation unit (130) may dynamically set an edge detection threshold or a binarization threshold on a frame-by-frame basis based on the shooting parameters. This reduces the instability of reference target detection caused by automatic exposure changes during night work.

[0036] In one embodiment of the present invention, the work image-based agricultural tool depth measuring device (10) may be capable of measuring the depth using only a single-camera-based work image. However, it is not limited thereto, and the image acquisition unit (230) may include multiple cameras to acquire work images from different angles. In this case, the calculation unit (130) may select the image in which the reference target is most stably detected among the multiple images. Alternatively, the calculation unit (130) may be configured to calculate the depth value by fusing the reference target positions of the multiple images.

[0037] When an image acquisition unit (230) including multiple cameras acquires working images from different angles, the operation unit (130) can perform reference target detection for each camera image and then select an image in which the reference target is most stably detected based on the reliability and temporal stability of the detection results. For example, the operation unit (130) may be configured to select an image with high reliability and small variation by comparing at least one of the following: whether detection was successful, the detection reliability score (such as the confidence score of deep learning-based Object Detection), the amount of variation (such as variance) of the representative pixel location of the reference target in the last N frames, or the fitting quality in edge-based detection (Hough transform cumulative value or fitting residual). Additionally, if the reliability of the selected image drops sharply or the representative pixel location changes rapidly, it may be switched to another camera image, and to reduce the rapid change in the reference value due to the switch, a method of evaluating candidate images in parallel for a certain frame interval and then switching may also be applied.

[0038] Additionally, the operation unit (130) may be configured to calculate a metacenter value by fusing the reference target positions of multiple images. In this case, the operation unit (130) may first convert the representative pixel position calculated from each camera image into the metacenter value of the corresponding camera using the camera-specific calibration result (metacenter scale line), and then perform fusion in the metacenter value space. As a fusion method, at least one of the following may be applied: a weighted average with weights proportional to the reliability score or the inverse of the variation amount; a median selection that is strong against false detection; or a method of averaging after excluding values ​​where the difference from other camera results exceeds a threshold value as outliers. For the fused metacenter value (or fused representative pixel position), the final metacenter value may be calculated through linear interpolation if necessary based on the zero reference line and the metacenter scale line.

[0039] FIG. 3 is an illustrative diagram for explaining a calibration unit (110) according to one embodiment of the present invention.

[0040] The work video-based agricultural implement depth measuring device (10) can measure the depth of an agricultural implement based on a work video acquired from a camera installed at the rear of a tractor. The work video may include the agricultural implement and the soil surface.

[0041] The calibration unit (110) can establish a correspondence relationship between pixel positions within the work image and actual meridian values ​​using a correction plate on which a preset pattern is formed. Additionally, the calibration unit (110) can perform the function of setting multiple meridian scale lines based on the established correspondence relationship. This calibration process can be performed under conditions identical to the actual work environment, that is, while maintaining the camera's installation position, shooting angle, focus, and field of view.

[0042] The calibration unit (110) can detect the pixel positions of the correction plate pattern from the work image. The calibration unit (110) can calculate how much a distance or displacement in the pixel coordinate system corresponds to as an actual physical length based on the actual dimension information of the pattern. This correspondence relationship may be a simple linear proportional equation, or a transformation that takes into account the perspective effect and tilt of the camera, for example, a homography-based transformation. Consequently, the calibration unit (110) can generate transformation information or a lookup table that causes the vertical pixel positions of the work image to correspond to the actual meridian values.

[0043] After the correspondence relationship is established, the calibration unit (110) can set multiple meridional scale lines defined by the pre-set meridional value intervals by mapping them to the image coordinate system. For example, if the meridional value interval is set to 5 cm, the calibration unit (110) can calculate pixel positions corresponding to each predetermined meridional value, such as +10 cm, +5 cm, 0 cm, and -5 cm. The calibration unit (110) can define the line corresponding to each calculated pixel position as a meridional scale line. The calibration unit (110) can store the information in memory so that the zero point setting unit (120) and the calculation unit (130) can refer to it in subsequent operations.

[0044] In one embodiment of the present invention, the pattern formed on the correction plate may have various forms that can be stably detected in an image. For example, the pattern of the correction plate may be a checkerboard pattern, a circular dot pattern, an ArUco marker, a QR code, a square grid pattern, a radial pattern, or a line pattern repeating at a predetermined interval. However, the present invention is not limited thereto. The pattern of the correction plate may have predetermined actual dimensions. This enables accurate conversion between pixel units and actual physical units.

[0045] In one embodiment of the present invention, the calibration unit (110) may include a quality verification process to improve the reliability of the calibration result. The calibration unit (110) may determine whether the pattern detection result satisfies a preset condition. For example, the calibration unit (110) may check whether the number of detected feature points is greater than or equal to a specific threshold. Here, a feature point is a point in a pattern formed on a calibration plate where the position on the image can be clearly defined, and may refer to a point representing the pattern, such as a grid corner point of a checkerboard pattern, a center point of a circular dot pattern, or a corner point of a marker pattern. Such feature points may be expressed in pixel coordinates, and the calibration unit (110) may calculate the correspondence between the pixel position and the actual dimension using the pixel coordinates of the feature points. In addition, the calibration unit (110) may also check whether the feature points are sufficiently distributed over the entire area of ​​the calibration plate (e.g., whether they are concentrated in a specific area or whether edge feature points are missing).

[0046] Alternatively, the calibration unit (110) may calculate a re-projection error and check whether the value is within an allowable range. For example, the calibration unit (110) may calculate the re-projection error by calculating the difference between the detected feature point location and the feature point location predicted by the calculated transformation information (e.g., correspondence relationship or transformation matrix), and may determine whether at least one of the average value, maximum value, or variance of the re-projection error satisfies a preset allowable condition. Additionally, when multiple frames are used, the calibration unit (110) may compare the re-projection error for each frame and exclude the frame with a relatively large error, or select the frame with the smallest error and apply it to the calibration.

[0047] If the quality verification result does not meet the standard, the calibration unit (110) may guide the user to retake the correction plate. Alternatively, the calibration unit (110) may ensure the accuracy of the calibration by automatically selecting a different video frame in the same working video section, or by sequentially evaluating multiple candidate frames and inducing the use of the frame with the best quality indicator.

[0048] FIG. 4 is an illustrative diagram for explaining a zero point setting unit (120) according to one embodiment of the present invention.

[0049] FIG. 4 illustrates, exemplarily, the process of the zero point setting unit (120) setting a zero point reference line. The zero point setting unit (120) can detect a specific part that includes at least one of the contour, corner, and surface boundary included in the upper structure of the agricultural implement (220) as a reference target from a work image in which the depth of the work at the start of the work is a preset initial value. The zero point setting unit (120) can set and store the pixel location of the detected reference target as a zero point reference line. At this time, the initial value may refer to the state in which the user sets the working depth of the agricultural implement (220) as a reference depth before the start of the work, and may be set, for example, to 0 cm. That is, the user may set the working depth to an initial value by adjusting the height or hydraulic pressure of the agricultural implement (220) and then perform zero point setting.

[0050] The zero point setting unit (120) can detect a reference target from a specific frame of the work video. For example, the zero point setting unit (120) can detect a reference target by selecting a video frame immediately after the work start button is input. For example, the reference target may be the top corner of the frame of the agricultural work machine (220), the outline of the reinforcement bar, or the boundary line between the upper structure and the background. However, the present invention is not limited thereto. When a reference target is selected, the zero point setting unit (120) can calculate a representative pixel location of the reference target. The representative pixel location may be defined as at least one of the average y-coordinate of the reference target outline, the y-intercept of the line fitting result, the y-coordinate of the bottom center point of the bounding box, or the y-coordinate of the lowest point of the mask outline.

[0051] The detection method of the reference target can be implemented in various ways. For example, the zero point setting unit (120) can detect edges in the region of interest using edge-based detection and determine a specific contour of the agricultural machine (220) as the reference target through Hough transform or line fitting. In another embodiment, the zero point setting unit (120) can detect the reference target using deep learning-based object detection. Additionally, the zero point setting unit (120) can use a hybrid method to find the approximate location of the reference target using deep learning object detection and to refine the boundary using edge-based detection to stably determine the representative pixel location. For example, the zero point setting unit (120) can first calculate the bounding box of the reference target using deep learning-based object detection to limit the region of interest of the reference target to the area around the bounding box, and then perform edge-based detection and line fitting within the limited region of interest to precisely calculate the representative pixel location of the reference target contour.

[0052] Meanwhile, the edge-based detection algorithm and the deep learning-based object detection model used in the zero-point setting unit (120) and the operation unit (130) may be the same or substantially identical algorithms and models capable of performing the same function.

[0053] The zero point setting unit (120) can set and store the calculated representative pixel position as the zero point reference line. The zero point setting unit (120) can specify the meridian scale line where the representative pixel position is located among the meridian scale lines set by the calibration unit (110), and define and store the specified meridian scale line as the zero point reference line corresponding to the meridian 0cm. The zero point reference line can be a reference corresponding to an initial value, for example, 0cm. The zero point reference line can be used as a reference line for the calculation unit (130) to calculate the meridian value. The plurality of meridian scale lines (-10cm, -5cm, reference meridian 0cm, +5cm, +10cm, +15cm) shown in FIG. 4 may indicate that the meridian scale lines set by the calibration unit (110) are mapped onto the work image. The zero point reference line can be set as the line corresponding to 0cm among these meridian scale lines.

[0054] In one embodiment of the present invention, the zero point setting unit (120) can perform reference target detection by setting a specific Region of Interest (ROI) rather than the entire image. The zero point setting unit (120) can set a pre-set range in the vertical direction from the zero point reference line, for example, a range of ±5cm or less, as the basic region of interest. The zero point setting unit (120) can limit the reference target detection range by setting the image area where the reference target mainly appears as the region of interest. After the zero point reference line is set, the range of ±5cm or less relative to the zero point reference line can be updated and used as the basic region of interest. The zero point setting unit (120) can be configured to perform edge detection or object detection within the corresponding region of interest. By limiting the region of interest, the amount of computation can be reduced, and false detections caused by background elements can be reduced.

[0055] In one embodiment of the present invention, the zero point setting unit (120) may store a zero point reference line and additionally store additional information for stabilizing the tracking of a reference target. Here, the additional information may refer to data or parameters that are directly used by the subsequent operation unit (130) to find (or track) the reference target again on a frame-by-frame basis and to adjust the detection parameters. For example, the zero point setting unit (120) may set a template area of ​​a pre-set size centered on the representative pixel position of the reference target in the working image at the time of zero point setting, and the template area may be defined by position information (e.g., top-left coordinate) and size information (e.g., width and height) in an image coordinate system. Additionally, the zero point setting unit (120) may store a pixel pattern included in the template area as template data, and the template data may include, for example, at least one of the raw pixel value of the template area (grayscale or specific channel pixel value), a normalized pixel value, or a feature calculated from the pixel value (e.g., gradient magnitude / direction, edge map, feature descriptor). As another example, the zero-point setting unit (120) may store the location and size of a region of interest (ROI) centered on the representative pixel location of the reference target, and the region of interest may be defined as a range that adds a pre-set margin around the reference target, serving as a search window to limit the search range of the reference target in subsequent frames. Additionally, the zero-point setting unit (120) may store at least one of the average luminance, standard deviation (contrast), and histogram distribution for the region of interest or template area at the time of zero-point setting as reference luminance statistics. This additional information may be used by the operation unit (130), described later, to reliably detect or track the reference target on a frame-by-frame basis and to preprocess or adjust threshold values ​​according to changes in lighting.

[0056] FIG. 5 is an illustrative diagram for explaining a calculation unit (130) according to one embodiment of the present invention.

[0057] Referring to FIG. 5, the operation unit (130) can detect the current pixel position of a reference target from an image frame of a work image acquired during the work. The operation unit (130) can calculate a hardness value corresponding to the current pixel position based on a zero reference line and a hardness scale line.

[0058] The operation unit (130) can perform reference target detection for each input image frame. Reference target detection can be performed within a Region of Interest (ROI) set by the zero point setting unit (120). For example, the operation unit (130) can set a pixel range corresponding to a pre-set range in the vertical direction from the zero point reference line, for example, ±5cm, as the region of interest. The operation unit (130) can perform reference target detection only within the set region of interest. The method of limiting the region of interest can reduce interference caused by the background. Limiting the region of interest can reduce the amount of computation compared to processing the entire image frame. The reduced amount of computation can improve processing speed.

[0059] The computation unit (130) can detect a reference target using an algorithm. For example, the computation unit (130) can use an edge-based detection technique. The computation unit (130) can apply a Gaussian filter to an image within a region of interest. The application of a Gaussian filter can reduce noise. The computation unit (130) can detect edges using a method such as Canny edge detection. The computation unit (130) can perform Hough Transform or Random Sample Consensus (RANSAC)-based line fitting on the detected set of edge pixels. The computation unit (130) can estimate a line segment corresponding to the contour of the upper structure of the agricultural machine (220) through line fitting. The computation unit (130) can calculate the location of the estimated line segment as the current pixel location of the reference target.

[0060] In one embodiment of the present invention, the computation unit (130) may utilize a deep learning-based object detection technique. The computation unit (130) may apply a detection model. The detection model may receive an image frame or a region of interest as input. The detection model may output a reference target in the form of a bounding box or a segmentation mask. For example, the detection model may be implemented in a structure including a backbone network, a feature neck, and a detection head. For example, the backbone network may be a ResNet family, a MobileNet family, a Cross Stage Partial (CSP) family, a Vision Transformer family, a ConvNeXt family, a Swin Transformer family, and an EfficientNet family. However, the present invention is not limited thereto.

[0061] For training the above detection model, training data may be constructed by collecting work video frames obtained from an actual work environment from a camera installed at the rear of the tractor (20). For example, the training data may be configured to include various conditions such as day and night, different lighting conditions, changes in soil color and texture, changes in the type and mounting status of the agricultural implement (220), changes in camera installation height / angle / distance, shaking or blurring caused by vibration, and partial obscuration caused by dust and splashing dirt. Additionally, the training data may include a correct answer label in the form of a bounding box or a segmented mask for a reference target (e.g., a specific part of the upper structure of the agricultural implement (220)) in each frame, and, if necessary, may also define and utilize the representative pixel location of the reference target (e.g., the center point at the bottom of the bounding box or a specific point on the outer edge of the mask) for training. As a training method, a fine-tuning approach using pre-trained weights as initial values ​​and the aforementioned training data can be applied. Additionally, to ensure robustness against lighting changes and noise, the model can be trained to minimize its loss function while applying data augmentation such as brightness / contrast changes, gamma correction, blur / noise addition, rotation / translation / scale transformation, and partial occlusion.

[0062] The current pixel position of the reference object can be calculated. The calculation unit (130) can calculate the meridian value based on the zero reference line and the meridian scale line. The current position of the reference object may be located above the zero reference line (reference line meridian 0cm). If the current position of the reference object is located above the zero reference line, the calculation unit (130) can calculate the current meridian as -5cm. The current position of the reference object may be located below the zero reference line. If the current position of the reference object is located below the zero reference line, the calculation unit (130) can calculate the current meridian as +5cm. The sign of the meridian value may be determined according to the definition of the system. In the present invention, if the pixel position of the reference object moves upward relative to the zero reference line, the meridian may be set to decrease. If the pixel position of the reference object moves downward relative to the zero reference line, the meridian may be set to increase.

[0063] The operation unit (130) can calculate the meridian value through linear interpolation when the current pixel position of the reference object is located between two adjacent meridian scale lines. Specifically, the operation unit (130) can perform interpolation based on the meridian value corresponding to each of the two adjacent meridian scale lines. For example, if the current pixel position is located between the 0cm scale line and the +5cm scale line, the relative ratio of the current position to the difference in pixel positions between the two scale lines can be calculated. The operation unit (130) can apply the calculated ratio to the difference in meridian values ​​between the two scale lines. The operation unit (130) can calculate the final meridian value through the applied result. The linear interpolation method can derive a continuous meridian value from discontinuous scale line information. The derived continuous meridian value can improve the precision of the measurement.

[0064] Meanwhile, in one embodiment of the present invention, the operation unit (130) can more stably detect or track a reference target in each image frame based on additional information stored by the zero point setting unit (120). For example, the operation unit (130) can reduce false detection and reduce the amount of computation by using the location and size of the stored region of interest (ROI) to limit the search range for the reference target in each frame to the region of interest, and by updating the region of interest based on the current pixel location of the reference target calculated in the previous frame. In addition, the operation unit (130) can calculate the similarity between the stored template data and the pixel pattern within the region of interest of the current frame (e.g., normalized correlation (NCC), absolute difference (SAD), or feature descriptor matching), determine the location where the similarity is maximum as the current pixel location of the reference target, or, if the deep learning-based object detection or edge-based detection result is unstable, use the template matching result as an auxiliary basis to determine the location of the reference target. Furthermore, the operation unit (130) can determine the degree of change in lighting by comparing the stored reference luminance statistics (average luminance, contrast, histogram, etc.) with the statistical value of the current frame, and if the change in lighting is large, it can perform preprocessing such as gamma correction, contrast normalization, and histogram correction, or improve the stability of reference target detection by adjusting the threshold of edge-based detection (e.g., edge detection threshold or binarization threshold). In addition, if the template matching similarity or object detection reliability score is below a preset threshold, the operation unit (130) can temporarily expand the region of interest or switch to another detection method to perform re-detection, and through this procedure, the interruption of calculating the central value due to failure to track the reference target can be reduced.

[0065] FIG. 6 is a flowchart illustrating the schematic flow of a method for measuring the depth of an agricultural machine based on a work video according to one embodiment of the present invention.

[0066] Referring to FIGS. 1 to 6, the method for measuring the plow depth of an agricultural implement based on a work image can be performed in a work image-based agricultural implement plow depth measuring device (10). The method for measuring the plow depth of an agricultural implement based on a work image can be based on a work image acquired from an image acquisition unit (230) installed at the rear of a tractor (20). The work image may include an agricultural implement (220) and a soil surface. The method for measuring the plow depth of an agricultural implement based on a work image can measure the plow depth of the agricultural implement (220).

[0067] A method for measuring the depth of a farming tool based on a working image, performed in a working image-based farming tool depth measuring device (10), may include a calibration step (S10), a zero point setting step (S20), and a calculation step (S30).

[0068] In step S10, the calibration unit (110) can set a plurality of meridian scale lines, which are correspondences between pixel positions in the work image and actual meridian values, using a correction plate on which a preset pattern is formed. The calibration unit (110) can detect pixel positions of the correction plate pattern from the work image. The calibration unit (110) can calculate a scale or transformation relationship between pixel positions and actual meridian values ​​based on the actual dimensions of the pattern. Subsequently, the calibration unit (110) can map and store the plurality of meridian scale lines on an image coordinate system according to preset meridian value intervals. If the position or angle of the camera changes or the agricultural machine (220) is replaced, the calibration unit (110) can re-perform the calibration step (S10) to update the scale line information.

[0069] In one embodiment of the present invention, the calibration step (S10) can detect the pixel location of a pattern from a working image. The calibration step (S10) can calculate the correspondence between the pixel location and the actual meridian value based on the actual dimensions of the pattern. The calibration step (S10) can set a plurality of meridian scale lines defined by pre-set meridian value intervals according to the correspondence.

[0070] In step S20, the zero point setting unit (120) can detect a specific part of the agricultural implement (220) as a reference target from a work image in which the depth of the work at the time of work start is a preset initial value. The zero point setting unit (120) can set and save the pixel location of the reference target as a zero point baseline. The user can perform zero point setting after setting the work depth of the agricultural implement (220) to an initial value before starting the work. The zero point setting unit (120) can detect a specific part of the agricultural implement (220) as a reference target in the frame at that time. The zero point setting unit (120) can save the representative pixel location of the detected reference target as a zero point baseline. Detection of the reference target can be performed using an edge-based detection or deep learning-based object detection method. The zero point setting unit (120) can set a specific area around the zero point baseline as a Region of Interest (ROI). The zero point setting unit (120) can reduce the amount of computation and reduce false detections by searching for reference targets only within the set area of ​​interest.

[0071] In one embodiment of the present invention, the zero point setting step (S20) may detect a specific part included in the upper structure of the agricultural machine (220) as a reference target. For example, the specific part may be a contour, a corner, a surface boundary, an area of ​​a specific color, an attached marker, a logo, or a bolt head. However, the present invention is not limited thereto. The zero point setting step (S20) may set and store the pixel location of the reference target as a zero point reference line.

[0072] In step S30, the operation unit (130) can detect the current pixel position of a reference target from an image frame of a work image acquired during the operation. The operation unit (130) can calculate a meridian value corresponding to the current pixel position based on a zero reference line and a meridian scale line. The operation unit (130) can determine the current position of the reference target while sequentially processing frames during the operation. If the current position of the reference target coincides with a specific meridian scale line, the operation unit (130) can output the corresponding meridian value. The calculated meridian value can be displayed through a display device, stored in a storage device, or transmitted to an external device through a communication interface.

[0073] In one embodiment of the present invention, the calculation step (S30) can calculate a meridian value when the current pixel position of a reference target is located between two adjacent meridian scale lines. The calculation step (S30) may be based on the meridian value corresponding to each of the two adjacent meridian scale lines. The calculation step (S30) can calculate a meridian value corresponding to the current pixel position through linear interpolation.

[0074] In one embodiment of the present invention, the calculation step (S30) may include a step of generating a work history by storing the calculated hardness value and the timestamp of the corresponding image frame together. The work history can be used to analyze changes in hardness by work section and to evaluate work quality.

[0075] FIG. 7 is a flowchart illustrating the detailed flow of an operation step according to one embodiment of the present invention.

[0076] Referring to FIGS. 6 and 7, the calculation step (S30) can be performed by the calculation unit (130). The calculation step (S30) can repeatedly process image frames of the work image acquired during the work. The calculation step (S30) may be a procedure for calculating and outputting a center value through the processed image frames.

[0077] In step S31, the operation unit (130) may input video frames. The operation unit (130) may sequentially receive video frames of the work video from the video acquisition unit (230). Along with the video frames, shooting parameters may be provided to the operation unit (130). The operation unit (130) may perform preprocessing on the input video frames. For example, preprocessing may include filtering to reduce noise in a low-light environment, or luminance and contrast normalization to respond to changes in lighting. In particular, when lighting conditions change in a night work environment, the operation unit (130) may detect a reference target after correcting contrast through gamma correction or local histogram equalization (CLAHE).

[0078] In step S32, the operation unit (130) can detect a reference target and calculate the current pixel location. The operation unit (130) can detect a reference target throughout the entire input image frame. To reduce the computational load, the operation unit (130) can prioritize the detection of the reference target within a set Region of Interest (ROI). The operation unit (130) can perform detection by using the Region of Interest stored by the zero point setting unit (120) or by setting a pre-set range from the zero point reference line as the Region of Interest. The operation unit (130) can also dynamically adjust the search range by updating the Region of Interest centered on the location of the reference target calculated in the previous frame.

[0079] In one embodiment of the present invention, step S32 may include a step of correcting camera shake caused by vibration. For example, the computation unit (130) may extract and match feature points between consecutive image frames. The computation unit (130) may correct image shake by estimating a transformation model based on the feature point matching results and warping the current frame. Alternatively, the computation unit (130) may estimate the amount of translation by performing phase correlation or block matching only within the region of interest. The computation unit (130) may perform reference target detection after correcting the region of interest through the estimated amount of translation.

[0080] In step S33, the calculation unit (130) can calculate a hardness value based on the zero reference line and the hardness scale line. Based on the current pixel position calculated in step S32, the calculation unit (130) can select two adjacent scale lines among a plurality of stored hardness scale lines. If the current pixel position matches a specific hardness scale line, the calculation unit (130) can calculate the hardness value corresponding to that hardness scale line as the final hardness value.

[0081] In one embodiment of the present invention, the calculation unit (130) can calculate a meridian value corresponding to the current pixel position through linear interpolation based on the meridian value corresponding to each of the two adjacent meridian scale lines when the current pixel position of the reference target is located between two adjacent meridian scale lines. The linear interpolation method can calculate an intermediate meridian value using the relative ratio of the current pixel position to the pixel distance between the two scale lines.

[0082] In one embodiment of the present invention, the calculation step (S30) can calculate a meridian value corresponding to the current pixel position through linear interpolation based on the meridian value corresponding to each of the two adjacent meridian scale lines when the current pixel position of the reference target is located between two adjacent meridian scale lines. Through this, the meridian value can be calculated as a continuous value without changing discontinuously.

[0083] In one embodiment of the present invention, step S33 may include the step of applying temporal filtering and outlier removal. For example, the computation unit (130) may apply a moving average filter or a Kalman filter to a sequence of current pixel positions calculated in each frame. The computation unit (130) may calculate the filtered positions and, by comparing with the previous estimated value, determine and remove positions that change rapidly beyond a preset threshold displacement as outliers. The computation unit (130) may improve measurement stability by calculating a metacenter value based on the filtered final positions.

[0084] In step S34, the calculation unit (130) may output a hardness value. For example, the output method may be a numerical display via a display device, an overlay display on a work video, log storage to a storage device, data transmission to an external controller, data transmission to a server, output of a warning sound via a notification generating device, or interaction with a Driver Information System. However, the present invention is not limited thereto. When outputting the hardness value, the calculation unit (130) may store or transmit additional information such as the timestamp of the current frame, the frame number, or shooting parameters together.

[0085] In step S35, the operation unit (130) can check whether the task is completed. Whether the task is completed can be determined based on the user's termination signal input, the receipt of a driving termination signal from the tractor (20), or the cessation of task video input from the video acquisition unit (230). If the task is not completed, the operation unit (130) can return to step S31 to process the next video frame and repeat the above procedures. If the task is completed, the operation unit (130) can terminate the operation step (S30).

[0086] The method for measuring the tillage depth of an agricultural implement based on work video according to the present invention can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination.

[0087] The program instructions recorded on the above-mentioned computer-readable recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software.

[0088] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, and flash memory.

[0089] Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0090] Although various embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0091] 1: Operation Video-Based Farm Tool Plowing Depth Measurement System 10: Work video-based agricultural machinery tillage depth measuring device 110: Calibration section 120: Zero setting section 130: Operation unit 20: Tractor 210: Tractor body 220: Farm machinery 230: Image acquisition unit

Claims

Claim 1 A work image-based agricultural implement depth measuring device for measuring the depth of an agricultural implement based on a work image including an agricultural implement and a soil surface, acquired from a camera installed at the rear of a tractor, comprising: a calibration unit for setting a plurality of depth scale lines that are corresponding to the relationship between a pixel position within the work image and an actual depth value, using a correction plate having a preset pattern formed thereon; a zero-point setting unit for detecting a specific part of the agricultural implement as a reference target from a work image in which the depth value at the start of the work is a preset initial value, and setting and storing the pixel position of the reference target as a zero reference line; and a calculation unit for detecting the current pixel position of the reference target from an image frame of a work image acquired during the work, and calculating a depth value corresponding to the current pixel position based on the zero reference line and the depth scale lines.The calibration unit includes detecting the pixel position of the pattern from the working image and calculating the correspondence relationship between the pixel position and the actual meridian value based on the actual dimensions of the pattern, and mapping and setting the plurality of meridian scale lines defined by a preset meridian value interval according to the correspondence relationship, wherein the correspondence relationship is generated as transformation information such that the vertical pixel position of the working image corresponds to the actual meridian value, and based on the transformation information, each of the plurality of meridian scale lines is defined and stored as a line passing through a pixel position corresponding to a different actual meridian value, and the calculation unit, when the current pixel position of the reference target is located between two adjacent meridian scale lines, calculates the meridian value corresponding to the current pixel position through linear interpolation based on the meridian value corresponding to each of the two adjacent meridian scale lines, wherein during the linear interpolation, the relative ratio of the current pixel position to the pixel distance between the two adjacent meridian scale lines is calculated, and the relative ratio is applied to the difference between the meridian values ​​corresponding to each of the two adjacent meridian scale lines to calculate the meridian corresponding to the current pixel position A work video-based agricultural machinery depth measuring device that calculates a value. Claim 2 delete Claim 3 A work image-based agricultural implement depth measuring device according to claim 1, wherein the zero point setting unit detects the specific part including at least one of the contour, edge, and surface boundary included in the upper structure of the agricultural implement as the reference target, and sets and stores the pixel position of the reference target as the zero point reference line. Claim 4 delete Claim 5 A method for measuring the plowing depth of a farming implement based on a work image-based farming implement plowing depth measuring device, wherein the plowing depth of the farming implement is measured based on a work image including the farming implement and the soil surface, acquired from a camera installed at the rear of a tractor, comprising: a calibration step of setting a plurality of plowing depth scale lines, which are corresponding relationships between pixel positions within the work image and actual plowing depth values, using a correction plate having a preset pattern formed thereon; a zero point setting step of detecting a specific part of the farming implement as a reference target from a work image in which the plowing depth value at the start of the work is a preset initial value, and setting and storing the pixel position of the reference target as a zero reference line; and a calculation step of detecting the current pixel position of the reference target from an image frame of a work image acquired during the work, and calculating a plowing depth value corresponding to the current pixel position based on the zero reference line and the plowing depth scale lines.The calibration step includes detecting the pixel position of the pattern from the working image, calculating the correspondence relationship between the pixel position and the actual meridian value based on the actual dimensions of the pattern, and mapping and setting the plurality of meridian scale lines defined by a preset meridian value interval according to the correspondence relationship, wherein the correspondence relationship is generated as transformation information such that the vertical pixel position of the working image corresponds to the actual meridian value, and, based on the transformation information, each of the plurality of meridian scale lines is defined and stored as a line passing through a pixel position corresponding to a different actual meridian value, and the calculation step, when the current pixel position of the reference target is located between two adjacent meridian scale lines, calculates the meridian value corresponding to the current pixel position through linear interpolation based on the meridian value corresponding to each of the two adjacent meridian scale lines, wherein during the linear interpolation, the relative ratio of the current pixel position to the pixel distance between the two adjacent meridian scale lines is calculated, and the relative ratio is applied to the difference between the meridian values ​​corresponding to each of the two adjacent meridian scale lines to calculate the meridian corresponding to the current pixel position A method for measuring the tillage depth of an agricultural implement based on work video, which calculates a value. Claim 6 delete Claim 7 In claim 5, the zero point setting step detects the specific part including at least one of the contour, corner, and surface boundary included in the upper structure of the agricultural implement as the reference target, and sets and stores the pixel position of the reference target as the zero point reference line, a method for measuring the depth of an agricultural implement based on an operation image. Claim 8 delete

Images