An agricultural farming activity video key frame extraction and uplink notarization method

Abstract

This invention relates to the field of smart agriculture video monitoring, specifically disclosing a method for extracting keyframes from agricultural activity videos and storing them on the blockchain. The method deploys edge computing devices near farmland cameras to collect and filter live video streams in real time. It generates candidate segments of agricultural activities using a sliding buffer and identifies harvesting behavior based on hand movement trajectories, contact relationships, and changes in object state, extracting high-evidence keyframes. Subsequently, it generates image-perceptual hashes, cryptographic hashes, device signatures, device identification information, and collection time information, which are written to the blockchain. The original keyframe image, corresponding short video segment, and background inspection data are stored in a distributed file system, achieving low-cost storage and reliable evidence storage of agricultural videos. This invention effectively distinguishes between similar actions misjudged based solely on a single frame's appearance, reducing the storage and transmission costs associated with long-term storage of all-day video.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture video monitoring technology, and more specifically, to a method for extracting key frames from agricultural activity videos and storing them on the blockchain. Background Technology

[0002] With the development of smart agriculture, digital agriculture, and the Internet of Things in agriculture, video equipment is increasingly being used in farmland to continuously monitor agricultural activities such as harvesting, fertilizing, spraying pesticides, pruning, and field inspections. Existing solutions typically use continuous 24 / 7 video recording for storage. While simple to implement, this method presents two significant practical problems: First, 24 / 7 video contains a large amount of empty scenes, repetitive footage, and low-quality images, which significantly increases storage and transmission burden over long-term storage. Second, even if keyframes are extracted, if the keyframe identification relies too heavily on the appearance of a single frame, misidentification between similar actions can easily occur, causing segments that don't need long-term storage to enter the evidence preservation process, further wasting storage resources.

[0003] In existing technologies, there are already solutions for using keyframe extraction in agricultural video processing. Chinese Patent Publication CN104021544A discloses a method and system for extracting keyframes from greenhouse vegetable disease monitoring videos, which extracts keyframes from disease monitoring videos using visual saliency and online clustering techniques. In 2014, Gao Lin published "Development and Experiment of Flowering Process Video Monitoring System Based on Keyframe Extraction Technology" in the journal *Transactions of the Chinese Society of Agricultural Engineering*. Figure 2 As shown, a method for reducing redundant information in videos of the flowering process is disclosed by extracting keyframes. In 2024, He Feng published "A Method for Recognizing Agricultural Behaviors in Facility Cucumbers Based on an Improved SlowFast Model" in the journal *Smart Agriculture*, indicating that for different agricultural behaviors, the agricultural behaviors described are as follows... Figure 3 As shown, introducing temporal information helps improve the accuracy of behavior discrimination in complex scenarios. The aforementioned publicly available information indicates that keyframe extraction, agricultural video analysis, and agricultural behavior recognition all have a foundation for development, but their technological focus is respectively on disease monitoring, plant process observation, and facility scene behavior classification, without simultaneously addressing the two problems of "misjudging similar actions in the field leading to incorrect keyframe extraction" and "achieving reliable keyframe evidence storage and low-cost long-term storage in synergy."

[0004] On the other hand, in the field of electronic data storage, the combination of blockchain and off-chain storage has been extensively studied. Existing technologies for tamper-proof detection of image and video content use perceptual hashing, authentication hashing, or content feature extraction to perform consistency checks on image and video frames, identifying whether images or videos have been replaced, edited, or tampered with. However, existing tamper-proof schemes are not integrated with the keyframe extraction process for agricultural activities, and cannot complete the continuous processing flow of "invalid image filtering—target behavior identification—keyframe extraction—trusted storage" at the edge. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of existing technologies, this invention provides a method for extracting keyframes from agricultural activity videos and storing them on the blockchain. This method involves deploying edge computing devices near farmland cameras and establishing a video sliding buffer to continuously process the farmland monitoring video stream, filtering invalid footage, and identifying picking and pruning actions based on hand movement trajectories, the contact relationship between the hand and the work object, and changes in the object's state after contact. Only candidate segments identified as target agricultural activities are extracted with high-evidence keyframes. Perceptual hashes, cryptographic hashes, device signatures, and time information are generated for each keyframe and written to the blockchain. The original image and corresponding short video segments are stored in a distributed file system, thus solving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for extracting keyframes from agricultural activity videos and storing them on the blockchain includes the following steps: collecting, filtering, extracting, and storing farmland field videos.

[0008] Step 1: Run an AI model on an edge computing device deployed near the farmland camera equipment to collect live video streams.

[0009] Step 2: The AI ​​model performs real-time filtering of images in the video stream, removing invalid images.

[0010] Step 3: Combine the effective footage with consecutive video data from the sliding buffer to generate candidate agricultural activity segments.

[0011] Step 4: Perform behavioral discrimination on candidate agricultural activity segments to distinguish between picking behavior and tidying up branches and leaves.

[0012] Step 5: Extract keyframes from candidate segments identified as picking behavior.

[0013] Step 6: Generate a digital evidence digest for the keyframe and write it to the blockchain for on-chain evidence storage.

[0014] Step 7: Store the original keyframe image, the corresponding candidate activity short video clip, and the relevant background inspection data in the distributed file system for off-chain storage.

[0015] As a further aspect of the present invention, step 1 involves running an AI model on an edge computing device deployed near the farmland camera to collect on-site video streams. This includes the following specific details: To ensure timely processing of farmland video near the acquisition point and avoid the transmission burden and processing delays caused by transmitting all raw video data back to a remote server 24 / 7, an edge computing device is deployed near the farmland camera, and a lightweight AI model is run on the edge computing device. The farmland camera continuously collects on-site video streams and sends the continuously acquired image data to the edge computing device in real time. The edge computing device processes the video stream continuously under limited computing power. Simultaneously, the edge computing device establishes a time-based, rolling-up sliding buffer for the video stream to temporarily store continuous video data within the most recent preset duration. This allows subsequent processing to not only acquire the current video frame but also retrieve continuous video content before and after the current moment.

[0016] As a further aspect of the present invention, step 2 involves the AI ​​model performing real-time screening of the images in the video stream to filter out invalid images. This includes the following specific content: The AI ​​model does not directly perform subsequent behavior recognition and keyframe extraction on all video streams. Instead, it first performs real-time screening of the images in the video stream to pre-remove images that do not have evidentiary value or do not have stable discrimination conditions, thereby reducing the pressure of subsequent processing.

[0017] The invalid images are categorized into four types: The first type is empty scene images, which only contain fields, furrows, trellises, roads, sky, or plant backgrounds, without including workers, tools, or objects being worked on. Although these images are part of normal farmland monitoring data, they do not directly reflect the occurrence of agricultural activities and therefore do not need to be included in the subsequent candidate activity segment generation process. The second type is images with low evidentiary value, which contain people, but these people are only shown walking, standing, patrolling, or observing, without forming effective interactions with the fruit area, branches and leaves area, or tools that can represent the target agricultural activity. Therefore, these images are not suitable as the basis for judging key agricultural activities. The third type is low-reliability images, which are difficult to reliably identify due to blurring, backlighting, occlusion, rain or fog interference, or abnormal brightness, resulting in unreliable behavior judgment results. The fourth type is redundant images, which are images whose changes compared to previous and subsequent frames are less than a set threshold and lack new evidentiary information. These images usually appear consecutively and consume computing resources without adding valid evidence.

[0018] As a further aspect of the present invention, step 3, which combines the effective images with the consecutive video data in the sliding buffer to generate candidate agricultural activity segments, includes the following specific content: For the effective images retained after the pre-screening, the key agricultural activities are not directly determined based on whether personnel, hands, or work objects appear in a single frame image. Instead, candidate agricultural activity segments are generated by combining the consecutive video data in the sliding buffer that is saved in a time-rolling manner, so as to avoid misjudgment due to insufficient information in a single frame.

[0019] Specifically, based on the target detection results and spatial relationship changes in the current effective frame, it is determined whether the candidate segment triggering conditions are met. The triggering conditions include: workers entering a preset work area, indicating that agricultural operations may be about to occur in that area; the distance between the hand area and the fruit area or branch and leaf area continuously decreasing, indicating a trend of approach and interaction; the appearance of the work tool area or storage area in the frame, indicating that the site may enter an actual work state; and the local frame motion amplitude exceeding a set threshold, indicating that there is a significant change in action in the relevant area. When the above four triggering conditions are met, continuous video data before and after the triggering time is retrieved from the sliding buffer and spliced ​​according to a preset time window to form a candidate agricultural activity segment, so that the segment simultaneously covers the action preparation stage, the action implementation stage, and the action result stage.

[0020] As a further aspect of the present invention, step 4 involves behavioral discrimination of candidate agricultural activity segments to distinguish between picking behavior and tidying behavior, including the following specific content: In response to the problem that "picking action" and "tidying action" are similar in a single frame image and are easily misjudged, the "picking action" and "tidying action" are distinguished based on the hand movement trajectory, the contact relationship between the hand and the work object, and the change in the state of the object after contact.

[0021] Candidate agricultural activity segments that have already been triggered and generated are retrieved from the sliding buffer, and these segments are divided into a continuous video frame sequence according to their chronological order. Each candidate agricultural activity segment includes at least a preparation phase before the action occurs, an implementation phase during the action, and a result phase after the action is completed.

[0022] In the obtained continuous video frame sequence, the following key regions are located and marked frame by frame: the hand region, the fruit region, the branch and leaf region, and the tidying region. The hand region represents the position where the worker actually performs the action; the fruit region represents the position of the object being picked; the branch and leaf region represents the position of the object being tidied; and the tidying region represents the location where the object might be moved after picking. After completing frame-by-frame localization, the spatial positions of the same target in adjacent frames are correlated to form a temporal position sequence of each key region in the entire candidate agricultural activity segment, namely, the temporal position sequence of the hand region, the temporal position sequences of the fruit and branch / leaf regions, and the temporal position sequence of the tidying region.

[0023] Based on the temporal position sequence of the hand region, the motion trajectory information of the hand in the entire candidate agricultural activity segment is extracted. The motion trajectory information includes at least: the direction of hand movement, displacement amplitude, movement speed, dwell position, and dwell duration. If the hand region gradually moves from a position far away from the work object towards the fruit or leaf area in consecutive frames, it indicates an active tendency to approach the work object; if the hand region shows a significant withdrawal after approaching, it indicates that the action may have completed some kind of interaction process.

[0024] Based on the hand movement trajectory information, and combined with the temporal position sequence of the fruit and leaf regions, it is determined whether a contact relationship has been formed between the hand region and the fruit or leaf region. The contact relationship includes at least: the approach start time, the contact occurrence time, the contact duration, and the contact end time. Specifically, when the spatial distance between the hand region and the fruit or leaf region continuously decreases and falls below a set contact threshold, and this state persists for several consecutive frames, a contact relationship is determined to have been formed; when the distance between the hand region and the work object region increases again, the contact relationship is determined to have ended.

[0025] Based on the established contact relationship, the states of the fruit area, leaf and branch area, and storage area before and after contact are compared, using the start and end times of contact as time boundaries, to extract object state change characteristics. These object state change characteristics include at least: whether the fruit target remains in its original attachment position; whether the fruit target has detached, been missing, or moved; whether the hand area exhibits a state of carrying the target object after contact ends; whether a new target object appears in the storage area; and whether the leaf and branch area only experiences localized swaying, changes in leaf position, or changes in occlusion relationships.

[0026] The hand movement trajectory information, contact relationship information, and object state change characteristics are uniformly mapped onto the temporal coordinate axis of the same candidate agricultural activity segment by a timestamp-based alignment method, thus forming a complete temporal evidence chain.

[0027] Based on the aforementioned temporal evidence chain, candidate agricultural activity segments are classified and judged to identify picking actions and tidying up branches and leaves. Specifically, when the temporal evidence chain shows that the hand area first approaches the fruit area continuously, then forms stable contact with the fruit area in consecutive frames, and in subsequent frames after the contact ends, it can be observed that the fruit detaches from its original attachment position, the fruit outline disappears from its original position, or shifts significantly, while the hand carries a target object corresponding to the appearance of the fruit during the withdrawal process, and a new target object appears in the storage area, it is determined that the complete result chain of "approaching the fruit - contacting the fruit - fruit detachment - target object transfer" is satisfied, and thus it is judged as a picking behavior; when the temporal evidence chain shows that the hand area first approaches the branch and leaf area or the area around the fruit covered by branches and leaves, then forms contact with the branch and leaf area, and in consecutive frames after the contact, only the branches and leaves swing locally, the leaf position changes, and the occlusion relationship changes, while the fruit remains in its original attachment position, the hand does not carry a target object after withdrawal, and no new target object appears in the storage area, it is determined that only the temporal chain of "approaching the branches and leaves - contacting the branches and leaves - changes in the branches and leaves" is satisfied, but the result chain of fruit detachment and transfer is not formed, and thus it is judged as a tidying behavior.

[0028] As a further aspect of the present invention, step 5 involves extracting keyframes from candidate segments identified as harvesting activities. This includes extracting high-evidence keyframes from candidate segments identified as harvesting activities (target agricultural activities). These high-evidence keyframes are not arbitrary video frames, but rather image frames that simultaneously characterize the worker, the object being harvested, and the state of the harvesting result. The state of the harvesting result includes at least one or more of the following: hand contact with fruit, fruit detachment from its original attachment position, hand retraction carrying the target object, and the transfer of the target object to the storage area. One or more high-evidence keyframes can be selected from the same candidate segment to enhance post-event verification capabilities.

[0029] Meanwhile, to reduce long-term storage costs, instead of storing all original videos around the clock for an extended period, only high-evidence keyframes, candidate short video clips of activities corresponding to the keyframes, and periodically extracted low-frequency background inspection data are stored for an extended period. For the full set of original videos, only short-term caching is performed, and the videos are automatically overwritten and deleted after the caching period expires.

[0030] As a further aspect of this invention, step 6 involves generating a digital evidence digest for the keyframes and writing it to the blockchain for on-chain evidence storage. This includes the following specific content: To prevent key images from being tampered with later, a digital evidence digest is generated for the extracted high-evidence keyframes. The digital evidence digest includes at least: an image-aware hash, a cryptographic hash, a device signature, device identification information, and acquisition time information. The image-aware hash is used to characterize the similarity of the main visual content of the image, adapting to scenarios where keyframes still need to be identified as the same content after slight compression, transcoding, or format conversion; the cryptographic hash is used to verify the byte-level integrity of the original file to detect whether the file has been directly tampered with; the device signature is used to prove that the keyframe was generated by the corresponding device; the device identification information is used to establish a mapping between the keyframe and the acquisition device; and the acquisition time information is used to establish the event occurrence sequence. Furthermore, the device private key can be stored in a secure chip, a trusted execution environment, or other protected key storage area, and the acquisition time information can be generated by a trusted clock source, a time synchronization service, or a verified device time to improve the credibility of the source and time information.

[0031] As a further aspect of this invention, step 7 involves storing the original keyframe image, the corresponding candidate activity short video clip, and related background inspection data in a distributed file system for off-chain storage. This includes the following specific steps: writing the digital evidence digest to the blockchain for on-chain evidence storage, and storing the original keyframe image, the corresponding candidate activity short video clip, and related background inspection data in the distributed file system for off-chain storage. Through the correspondence between the on-chain digest and the off-chain file address information or content addressing identifier, collaborative on-chain and off-chain storage is achieved. If someone subsequently attempts to replace the keyframe, modify the file content, forge the source, or tamper with the time-related information, the anomaly can be detected through cryptographic hash verification, device signature verification, and comparison of on-chain and off-chain records.

[0032] The technical effects and advantages of this invention, a method for extracting and storing keyframes from agricultural activity videos, are as follows: This invention deploys edge computing devices near farmland cameras to process farmland monitoring video streams in real time. It first filters out empty scenes, low-value, low-reliability, and redundant footage. Then, it combines consecutive video data from a sliding buffer to generate candidate agricultural activity segments. Based on hand movement trajectories, the contact relationship between the hand and the work object, and changes in the object's state after contact, it constructs a temporal evidence chain. This effectively distinguishes between harvesting and pruning, avoiding misjudgments based solely on single-frame appearances, and ensuring that keyframes with true evidentiary value are extracted. The footage is accurately extracted. Simultaneously, this invention extracts high-evidence keyframes only from candidate segments identified as target agricultural activities. The image-perceptual hash, cryptographic hash, device signature, device identification information, and acquisition time information of the keyframes are written into the blockchain. The original keyframe image, corresponding short video clip, and background inspection data are stored in a distributed file system, achieving collaborative storage on and off the blockchain. Therefore, it not only significantly reduces the storage and transmission costs associated with long-term, 24 / 7 video preservation but also improves the accuracy of agricultural activity identification and keyframe extraction, and enhances the credibility of key evidence sources, content integrity, post-verification capabilities, and anti-tampering monitoring capabilities. Attached Figure Description

[0033] Figure 1 This is a flowchart of a method for extracting keyframes from agricultural activities videos and storing them on a blockchain, according to the present invention.

[0034] Figure 2 This is a schematic diagram of video monitoring based on keyframe extraction technology, a current technology.

[0035] Figure 3 The image shows the video frame results of detecting different cucumber farming behaviors using existing technologies.

[0036] Figure 4 This is a schematic diagram of the base monitoring system of the present invention.

[0037] Figure 5 This is a schematic diagram of the agricultural products uploaded to the blockchain according to the present invention.

[0038] Figure 6 Clicking on a specific agricultural product will redirect you to a detailed illustration of the agricultural product.

[0039] Figure 7 This is a schematic diagram illustrating the core statistical data of the platform's data upload to the blockchain, as shown in this invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1

[0042] like Figure 1 As shown, a method for extracting keyframes from agricultural activity videos and storing them on the blockchain includes the following steps:

[0043] Step 1: Run an AI model on an edge computing device deployed near the farmland camera equipment to collect live video streams.

[0044] Step 2: The AI ​​model performs real-time filtering of images in the video stream, removing invalid images.

[0045] Step 3: Combine the effective footage with consecutive video data from the sliding buffer to generate candidate agricultural activity segments.

[0046] Step 4: Perform behavioral discrimination on candidate agricultural activity segments to distinguish between picking behavior and tidying up branches and leaves.

[0047] Step 5: Extract keyframes from candidate segments identified as picking behavior.

[0048] Step 6: Generate a digital evidence digest for the keyframe and write it to the blockchain for on-chain evidence storage.

[0049] Step 7: Store the original keyframe image, the corresponding candidate activity short video clip, and the relevant background inspection data in the distributed file system for off-chain storage.

[0050] This embodiment uses a greenhouse tomato growing area as an example to illustrate the complete implementation process of the present invention. Several farmland camera devices are installed along the planting path on-site, such as… Figure 4 The diagram illustrates a farmland monitoring scenario using video cameras in this embodiment. Each camera continuously captures video streams from its corresponding planting row, with video resolutions ranging from 1280×720 to 1920×1080 and frame rates from 15fps to 30fps. An edge computing device is positioned near each camera. This edge computing device employs a low-power hardware platform with an embedded CPU and AI acceleration unit, enabling it to complete the method steps of this invention under limited computing power and bandwidth conditions. The edge computing device communicates with blockchain nodes and the distributed file system via wired or wireless networks.

[0051] Farmland camera equipment continuously collects live video streams and sends them to an edge computing device. The edge computing device internally establishes a sliding buffer, which uses a circular queue to store the most recently viewed video data for a preset duration. The buffer duration can be set from 10 seconds to 180 seconds; in this embodiment, it is set to 60 seconds. The purpose of using a sliding buffer is that when the system detects suspected agricultural activity, it can not only acquire several video frames at the time of the activity but also simultaneously extract continuous context segments before and after the activity, thus avoiding behavior recognition based solely on a single frame.

[0052] In this embodiment, the sliding buffer does not persist all videos long-term, but only maintains short-term caches. When a new video frame arrives, older video frames that exceed the cache length are automatically overwritten and deleted. In this way, the entire set of original videos does not accumulate over a long period of time, suppressing storage growth at the source. At the same time, once a subsequent step determines that a candidate segment belongs to the target agricultural activity, the corresponding consecutive video segments before and after the candidate segment will be retrieved from the sliding buffer and transferred to the long-term storage process.

[0053] The lightweight AI model first performs invalid image filtering on the video stream. The invalid image filtering unit filters images from four dimensions.

[0054] The first dimension is target existence filtering. The invalid image filtering unit detects the personnel area, work tool area, and work object area in the current video frame. If there are no workers and no valid combination of work tools and work objects in the image, the image is determined to be an empty scene. In this embodiment, an empty scene includes, but is not limited to, images that only capture planting furrows, trellises, sky, roads, foliage backgrounds, or vacant plots.

[0055] The second dimension is the screening of evidence value. If workers are detected in the image, but they only stand, walk, patrol, pause briefly, or observe the growth, without any effective interaction with the fruit area, leaf area, or tool area, then the image is judged as having low evidence value. The reason for this is that this invention aims to solve the problem of "keyframe extraction of key agricultural activities," rather than saving all images showing people.

[0056] The third dimension is credibility screening. The invalid image filtering unit detects the image's sharpness, brightness, contrast, and occlusion ratio. If the image is severely blurry, has excessive backlighting, fruit or hands are largely obscured, rain or fog causes loss of image detail, or the image is too dark or overexposed, then the image is judged as a low-credibility image. Even if low-credibility images enter the subsequent judgment process, the risk of misjudgment increases due to insufficient evidence; therefore, they are filtered out in advance in this invention.

[0057] The fourth dimension is redundancy filtering. The invalid image filtering unit calculates the change range between the current frame and the previous frame or several previous frames. When the change range between multiple consecutive frames is consistently lower than a set threshold, these frames are judged as duplicate redundant images. In this embodiment, thresholds can be set for frame difference, optical flow statistics, target position change, or comprehensive motion score to eliminate consecutive frames that are static for a long time or have almost no new information.

[0058] After the above four-dimensional screening, only valid footage is sent to the next step. The advantage of this setting is that it not only filters out empty shots, but also filters out a large number of non-target shots that are not suitable as evidence, making the subsequent candidate segment generation more focused.

[0059] Candidate agricultural activity segments are generated from the filtered valid footage. A candidate segment is not a single frame, but a time window containing information about the events before, during, and after the action. In this embodiment, candidate segments are triggered when the following conditions are met:

[0060] 1. Personnel have been detected entering the preset work area;

[0061] 2. A continuous approach from the hand area to the fruit area or the leaf area was detected;

[0062] 3. The storage container area, picking basket area, or cutting tool area were detected in the screen;

[0063] 4. If the amplitude of local motion exceeds the trigger threshold, continue to set the number of frames.

[0064] When the triggering condition is met, the candidate agricultural activity segment generation unit retrieves the preceding video data before the triggering moment and the following video data after the triggering moment from the sliding buffer, and splices them into a candidate agricultural activity segment. In this embodiment, the candidate segment duration is set to 6 seconds, with 2 seconds before the triggering point (triggering moment) and 4 seconds after the triggering point (triggering moment). The purpose of this is to retain information about the action preparation stage, the action implementation stage, and the action result stage, so that the candidate segment can provide a complete chain of evidence for subsequent short-sequence behavior discrimination.

[0065] Behavioral discrimination is performed on candidate agricultural activity segments. This embodiment mainly addresses the problem of misjudgment caused by the similar appearance of "picking actions" and "pruning actions" in a single frame image in a field environment. Based on the hand movement trajectory, the contact relationship between the hand and the work object, and the change in the object's state after contact, short-term behavioral discrimination is performed on candidate agricultural activity segments:

[0066] First, candidate agricultural activity segments that have already been triggered and generated are retrieved from the sliding buffer, and these segments are then divided into a continuous video frame sequence according to their chronological order. Each candidate agricultural activity segment includes at least a preparation phase before the action occurs, an execution phase during the action, and a result phase after the action is completed. By retaining continuous video data before and after the action, the subsequent judgment process no longer relies on static judgments based on single-frame images, but can utilize the complete temporal context to analyze the behavioral evolution process, thus providing a continuous data foundation for subsequent extraction of motion trajectories, establishment of contact relationships, and recognition of changes in result states.

[0067] In the continuous video frame sequence, key regions are located and marked frame by frame. These key regions include at least: a hand region, a fruit region, a branch and leaf region, and a storage region. Specifically, the hand region represents the position where the worker actually performs the action, the fruit region represents the position of the object being picked, the branch and leaf region represents the position of the object being sorted, and the storage region represents the location where the object might be moved after picking. After completing frame-by-frame localization, the spatial positions of the same target in adjacent frames are correlated and associated to form a temporal position sequence of each key region within the candidate agricultural activity segment. This sequence specifically includes the temporal position sequences of the hand region, fruit region, branch and leaf region, and storage region.

[0068] Based on the temporal position sequence of the hand region, the motion trajectory information of the hand in the entire candidate agricultural activity segment is extracted. The motion trajectory information includes at least the direction of hand movement, displacement amplitude, movement speed, dwell position, and dwell duration. In specific implementation, the coordinates of the hand center point can be extracted in consecutive frames, the displacement vector and displacement length between adjacent frames can be calculated according to the frame sequence, and the dwell time of the hand in a specific area can be counted. When the spatial distance between the hand region and the fruit region or branch and leaf region continuously decreases within 3 to 8 consecutive frames, preferably continuously decreasing within 5 consecutive frames, it is determined that the hand region has an active tendency to approach the work object; if the hand region shows obvious withdrawal after approaching, it indicates that the action has entered the recovery stage after the interaction is completed. It can be seen that the hand motion trajectory information can not only reflect whether the hand initiates the work action, but also provide a preliminary basis for subsequent judgment on whether contact has actually occurred.

[0069] Based on the hand movement trajectory information, and combined with the temporal position sequence of the fruit area and the temporal position sequence of the branch and leaf area, it is determined whether a contact relationship has been formed between the hand area and the fruit area or the branch and leaf area. The contact relationship includes at least the approach start time, the contact occurrence time, the contact duration, and the contact end time. Specifically, when the spatial distance between the hand area and the fruit area or the branch and leaf area continuously decreases and falls below a set contact threshold, and this low-distance state persists for 2 to 5 consecutive frames, preferably for 3 consecutive frames, then a contact relationship is determined to have been formed between the hand area and the corresponding work object area; when the distance between the hand area and the work object area increases again and returns to a non-contact state, the contact relationship is determined to have ended. Through this method, the system can further refine the process of "hand approaching the work object" into a complete interactive process of "approach-contact-departure," and clearly define the time boundary of the contact occurrence, thereby providing an accurate dividing point for comparing the states before and after contact.

[0070] Based on the established contact relationship, the states of the fruit area, branch and leaf area, and storage area before and after contact are compared, with the start and end times of contact as time boundaries, in order to extract the characteristics of the object's state changes. Specifically, the system uses several frames before the start of contact as the pre-contact state and several frames after the end of contact as the post-contact state. For the fruit region, the system records the original attachment position, outline range, and positional relationship between the fruit target and the surrounding reference background before contact. After contact, it determines whether the fruit target is still at its original attachment position. If the fruit outline at the original position disappears, shrinks significantly, or deviates from the original reference position, the system extracts the fruit detachment, missing, or position transfer features. For the branches and leaves region, the system compares the leaf outline distribution, occlusion range, and local swing amplitude before and after contact. If only the branches and leaves swing, the leaves deflect, or the occlusion relationship changes, and the original attachment position of the fruit does not disappear or transfer, the system extracts the branches and leaves change features. For the hand region and the storage region, the system further determines whether a new target object corresponding to the appearance of the fruit appears in the hand region after contact and whether a new target object appears in the storage region. If the hand carries a target object when it withdraws, or a new target object appears in the storage region after contact, the system extracts the target object transfer features.

[0071] The hand movement trajectory information, contact relationship information, and object state change characteristics are uniformly mapped onto the temporal coordinate axis of the same candidate agricultural activity segment by a timestamp-based alignment method, thus forming a complete temporal evidence chain.

[0072] Based on the aforementioned temporal evidence chain, candidate agricultural activity segments are classified and judged to distinguish between harvesting behavior and pruning behavior. Specifically, when the temporal evidence chain shows that the hand area first continuously approaches the fruit area, and then forms stable contact with the fruit area in 2 to 5 consecutive frames, with a preference for forming stable contact with the fruit area in 3 consecutive frames, and in subsequent frames after the contact ends, it can be observed that the fruit detaches from its original attachment position, the fruit outline disappears or shifts significantly from its original position, and the hand carries a target object corresponding to the appearance of the fruit during the withdrawal process, and a new target object appears in the storage area, the system determines that the current candidate agricultural activity segment satisfies the complete result chain of "approaching the fruit - contacting the fruit - fruit detachment - target object transfer", and thus judges it as harvesting behavior. Correspondingly, when the temporal evidence chain shows that the hand area first approaches the branch and leaf area or the area around the fruit covered by the branch and leaf, and then makes contact with the branch and leaf area, and in the continuous frames after the contact, only the branch and leaf swings locally, the leaf position changes, and the occlusion relationship changes, the fruit remains in its original attachment position, the hand does not carry the target object after withdrawing, and no new target object appears in the storage area, the system determines that the current candidate agricultural activity segment only satisfies the temporal chain of "approaching the branch and leaf - contacting the branch and leaf - changes in the branch and leaf", and does not form the result chain of fruit detachment and transfer, thus determining it as a branch and leaf tidying behavior.

[0073] High-evidence keyframes are extracted only from candidate segments identified as the target agricultural activity. A high-evidence keyframe is an image frame that simultaneously possesses two or more of the following elements and whose overall image quality meets the requirements: 1. A clearly visible worker or hand area; 2. A clearly visible object area, including fruit or other target agricultural products; 3. A clearly visible result state, including fruit detachment, hand retraction carrying the target object, or the target object entering the storage area. In this embodiment, an evidence score is calculated for each frame in the candidate segments. The evidence score is a combination of contact clarity, result state visibility, subject integrity, and image quality. The frame with the highest score is selected as the first keyframe. To strengthen the evidence chain, another frame with the second highest score but a different time position can be selected as the second keyframe, thus forming a combined evidence of "contact keyframe + result keyframe".

[0074] In this embodiment, the storage objects are managed in different categories. The first category consists of all original videos available 24 / 7. These videos are only stored in a sliding buffer and are automatically overwritten and deleted after the buffer expires, without being stored in long-term storage.

[0075] The second category is short video clips of candidate agricultural activities. These clips are only included in the long-term storage process when they are identified as the target agricultural activity by short-time behavior discrimination. In this embodiment, the storage period can be set to 30 to 365 days.

[0076] The third category is original keyframe images with high evidentiary value. This type of data belongs to long-term core storage objects, and the storage time can be the same as that of candidate fragments, or even longer.

[0077] The fourth category is low-frequency background inspection data. This type of data is not used to record every agricultural activity, but rather to assist in subsequent spatiotemporal consistency verification. In this embodiment, a set of background inspection images is extracted every 10 minutes, with each set containing 1 to 3 frames.

[0078] In this embodiment, a digital evidence digest is also generated for high-evidence keyframes. The digital evidence digest includes: 1. an image-aware hash, reflecting the main visual content of the keyframe; 2. a cryptographic hash, reflecting the byte-level integrity of the original keyframe file; 3. a device signature, proving that the keyframe was generated by the corresponding device; 4. device identification information, used to locate the data source; 5. acquisition time information, used to establish event sequence; and 6. off-chain file address information or content addressing identifier, used to establish on-chain and off-chain mapping. In this embodiment, the image-aware hash is calculated using a combination of pHash and dHash, and the cryptographic hash uses SHA-256. The device signature uses the device's private key to sign the above digest fields. The device's private key is stored in the secure chip of the edge computing device. The acquisition time information is periodically synchronized with the time synchronization service through the edge computing device, and an abnormal state is recorded when time synchronization fails. Digital evidence digests are written to blockchain nodes, forming on-chain records that cannot be arbitrarily altered. High-evidence keyframe original images, corresponding candidate short video clips, and low-frequency background inspection data are uploaded to a distributed file system, and off-chain file address information or content addressing identifiers are obtained. Subsequently, this off-chain identifier is written into the on-chain record, thus forming a correspondence between the on-chain digest and the off-chain original text. To facilitate viewing of agricultural product-related evidence information after on-chain registration, this embodiment allows the display of basic information of on-chain objects and corresponding on-chain records through an on-chain agricultural product information display page, such as... Figure 5 As shown. After displaying the list of on-chain agricultural product information, you can also select a specific agricultural product to enter its corresponding details page, where you can further view the detailed information of that agricultural product and its associated evidence storage content, such as... Figure 6 As shown.

[0079] After the keyframe digital evidence digest is written to the blockchain and the on-chain and off-chain correspondence is established, to facilitate the overall display and query of the on-chain evidence storage operation, a statistics display page can be used to centrally present blockchain operation information such as the number of on-chain entries for the day, the number of on-chain entries for the week, the cumulative total number of on-chain entries, block height, and network status. Figure 7As shown, when a keyframe needs to be verified later, the original image can be retrieved from off-chain, its cryptographic hash and perceptual hash can be recalculated, and compared with the on-chain record; at the same time, it can verify whether the device signature is valid, whether the time information is consistent, and whether the off-chain file address information matches. If any verification fails, the system will generate an anomaly alarm.

[0080] In this embodiment, spatiotemporal consistency auxiliary verification is performed on off-chain stored data. It should be noted that spatiotemporal consistency auxiliary verification is not a standalone final tampering criterion, but rather an additional mechanism used to assist in screening for risks of abnormal replacement, splicing, or forgery.

[0081] In this embodiment, several fixed reference areas are pre-selected in the video footage, including trellis connection points, uprights, camera mounting bases, and ground markers. These areas are chosen because they are relatively stable and do not change drastically with normal agricultural activities like fruits and leaves. Features are extracted from the same fixed reference areas in background inspection images at different time points, and normalized comparisons are performed using time period labels, weather status labels, and season labels. If unreasonable abrupt changes are found in the structural outline, spatial relationship, or stable texture features of the fixed reference areas, and these changes cannot be explained by weather, lighting, or seasonal changes, the system marks the relevant downstream data as data requiring consistency verification.

[0082] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0083] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An agricultural farming activity video key frame extraction and uploading and notarization method, which collects, screens, extracts and notarizes field video, characterized in that, Includes the following steps: Step 1: Run an AI model on an edge computing device deployed near the farmland camera equipment to collect live video streams; Step 2: The AI ​​model performs real-time filtering of images in the video stream, removing invalid images; Step 3: Combine the effective footage with consecutive video data from the sliding buffer to generate candidate agricultural activity segments; Step 4: Perform behavioral discrimination on candidate agricultural activity segments to distinguish between picking behavior and tidying up branches and leaves; Step 5: Extract keyframes from candidate segments identified as picking behavior; Step 6: Generate a digital evidence digest for the keyframe and write it to the blockchain for on-chain evidence storage. Step 7: Store the original keyframe image, the corresponding candidate activity short video clip, and the relevant background inspection data in the distributed file system for off-chain storage; In step 4, candidate agricultural activity segments are divided into video frame sequences according to chronological order. Key areas are located and marked frame by frame to form temporal position sequences of hand area, fruit area, branch and leaf area, and tidying area. Based on the temporal position sequence of hand area, the motion trajectory information of hand in candidate agricultural activity segments is extracted. Combined with the temporal position sequences of fruit area and branch and leaf area, the contact relationship between hand area and fruit area and branch and leaf area is determined. The time boundary is the start time and end time of contact. The object state change features are extracted to form a temporal evidence chain to distinguish picking action and tidying action.

2. The agricultural farming activity video key frame extraction and uploading and notarizing method according to claim 1, characterized in that, The distinction between picking and tidying actions includes: when the sequential evidence chain shows that the hand area first approaches the fruit area and then makes contact with the fruit area in consecutive frames, and after the contact ends, it is observed that the fruit detaches from its original attachment position, the outline of the fruit at its original position disappears, and the hand carries a target object corresponding to the appearance of the fruit during the withdrawal process, and a new target object appears in the storage area, it is determined to be a picking action; when the sequential evidence chain shows that the hand area first approaches the fruit periphery area of ​​the branch and leaf area and makes contact with the branch and leaf area, and after the contact, only the branch and leaf swings locally, the leaf position changes and the occlusion relationship changes in consecutive frames, the fruit remains in its original attachment position, the hand does not carry a target object after withdrawal, and no new target object appears in the storage area, it is determined to be a tidying action.

3. The method of claim 2, wherein, Keyframes are extracted from candidate segments identified as picking activities. The keyframes refer to image frames that can characterize the status of the workers, the objects being picked, and the results of the work.

4. The agricultural farming activity video key frame extraction and uploading and notarizing method according to claim 3, characterized in that, A digital evidence digest is generated for the key frame to prevent the key image from being tampered with later. The digital evidence digest includes at least: image perceptual hash, cryptographic hash, device signature, device identification information, and acquisition time information.

5. The agricultural farming activity video key frame extraction and uploading and notarizing method according to claim 4, characterized in that, The digital evidence digest is written to the blockchain for on-chain evidence storage, and the original image of the key frame, the corresponding candidate video clips of the activity, and the relevant background inspection data are stored in the distributed file system for off-chain storage. Through the correspondence between the on-chain digest and the off-chain file address information or content addressing identifier, on-chain and off-chain collaborative storage is achieved.

6. The method of claim 1, wherein, The invalid images include at least one of the following: empty scene images, which only contain fields, furrows, trellises, roads, sky, or plant backgrounds but do not contain workers, tools, or objects being worked on; low evidence value images, which contain people but they are only in a walking, standing, patrolling, or observing state and do not interact effectively with the fruit area, leaf area, or tools being worked on; low confidence images, which cannot reliably identify hands, objects being worked on, or details of local movements due to blurring, backlighting, occlusion, rain or fog interference, or abnormal brightness; and redundant images, which have a change in magnitude less than a set threshold compared to the previous and next frames and lack new evidence information.

7. The method of claim 1, wherein, The trajectory information of the hand in the candidate agricultural activity segment includes at least the direction of movement, displacement amplitude, movement speed, stopping position, and stopping duration of the hand.

8. The method for extracting keyframes from agricultural activity videos and storing them on the blockchain according to claim 1, characterized in that, The construction of the temporal evidence chain includes: using a timestamp-based alignment method to uniformly map hand movement trajectory information, contact relationship information, and object state change characteristics onto the temporal coordinate axis of the same candidate agricultural activity segment.

9. The method for extracting keyframes from agricultural activity videos and storing them on the blockchain according to claim 1, characterized in that, The farmland camera continuously collects on-site video streams and sends the continuously acquired image data to the edge computing device in real time.

10. The method for extracting keyframes from agricultural activity videos and storing them on the blockchain according to claim 9, characterized in that, The edge computing device establishes a sliding buffer for the video stream that is updated on a time-based basis, temporarily storing continuous video data within a preset duration. This enables subsequent processing to not only obtain the video frame at the current moment, but also to retrieve continuous video content before and after the current moment.

Images