An intelligent monitoring system based on internet of things machine vision

Abstract

The application relates to the technical field of Internet of Things intelligent monitoring, and discloses an intelligent monitoring system based on Internet of Things machine vision, which comprises the following modules: an agricultural camera, an edge computing terminal, an Internet of Things gateway, a server and an early warning execution device. The agricultural camera collects leaf surface images and mothboard images of monitoring points. The edge computing terminal determines the effective state of the images, extracts a suspected pest damage area when the definition condition and the monitoring object completeness condition are met, and generates a supplementary collection instruction when the conditions are not met. The Internet of Things gateway sends the suspected pest damage area or the supplementary collection instruction to the server. The server associates the leaf surface images and the mothboard images, determines a pest damage early warning level, sends a collection adjustment instruction to the agricultural camera, and sends an early warning instruction to the early warning execution device. Through effective state determination, image association and collection feedback control, the pest damage early warning level corresponds to the image state of the monitoring point and the subsequent collection process.

Description

Technical Field

[0001] This invention relates to the field of IoT intelligent monitoring technology, and in particular to an intelligent monitoring system based on IoT machine vision. Background Technology

[0002] The field of IoT-based intelligent monitoring technology typically revolves around the acquisition, remote transmission, identification, and information display of the monitored objects' status. It involves the collaborative acquisition of on-site images and environmental conditions by camera devices, sensors, edge computing terminals, IoT gateways, servers, and terminal devices, with monitoring information transmitted via wired or wireless networks. Traditional IoT-based machine vision-based intelligent monitoring systems refer to systems that identify and issue early warnings for insect bodies, eggs, insect spots, and damaged leaves in farmland, greenhouses, or orchards. These systems typically involve agricultural cameras, insect-attracting boards, insect monitoring points, edge computing terminals, IoT gateways, servers, display terminals, and on-site alarms. Agricultural cameras acquire images of leaves or insect-attracting boards at set intervals. Edge computing terminals perform basic image preprocessing and target feature extraction. The server combines insect samples and early warning conditions to determine the insect type, number of insects, and damage level, and sends the judgment record to the display terminal or on-site alarm.

[0003] In agricultural pest monitoring, agricultural cameras collect leaf images or insect-attracting board images at fixed intervals. Edge computing terminals mainly complete basic image preprocessing. The server provides pest type and damage level based on pest samples and early warning conditions. When a single image is affected by blurring or missing monitoring objects, the insect information and leaf damage information are separated. The pest records formed at different collection times are not well connected, the early warning judgment criteria tend to be singular, and there is a lack of reverse adjustment between the collection end and the early warning output end. Subsequent supplementary collection and on-site alarms are difficult to correspond to the status of the monitoring point. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent monitoring system based on Internet of Things machine vision, used in the process of monitoring agricultural pests to perform edge-side validity determination, extraction of suspected pest areas, server-side correlation fusion, and collection feedback control on leaf images and insect-attracting board images at the same monitoring point. This enables the pest warning level to be formed by combining the effective status of the image, the distribution of insect spots, the count of insects, and historical pest records, reducing the impact of single image or fixed threshold determination on pest warnings, and ensuring that the supplementary collection, model parameter adjustment, collection frequency adjustment, and warning execution processes correspond to the status of the monitoring point.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an intelligent monitoring system based on Internet of Things (IoT) machine vision, comprising an agricultural camera, an edge computing terminal, an IoT gateway, a server, and an early warning execution device.

[0006] Agricultural cameras are used to collect leaf and insect-attracting board images from the same monitoring point and send the images, along with the monitoring point identifier and collection time, to an edge computing terminal. When an agricultural camera collects leaf and insect-attracting board images from the same monitoring point, it records the camera's installation location, the crop planting area, and the insect-attracting board's placement location as a monitoring point identifier in the captured image. After receiving the captured images, the edge computing terminal creates image processing records using the monitoring point identifier as an index. Images with the same monitoring point identifier are grouped into the same monitoring point image group, while images with different identifiers have separate image processing records created for each group.

[0007] The edge computing terminal utilizes a pre-deployed lightweight semantic segmentation network to synchronously process received leaf and insect-attracting board images locally, extracting image regions within the leaf and insect-attracting board boundaries. The edge computing terminal determines the image's validity based on the integrity of the leaf and insect-attracting board boundaries and the clarity of the extracted image regions. When the image's validity meets a preset clarity threshold and both the leaf and insect-attracting board boundaries are within the valid image area, the edge computing terminal extracts suspected pest-infested areas containing insect spots or insects from the image region based on the semantic segmentation network's output. When the image's validity does not meet the preset clarity threshold or the leaf and insect-attracting board boundaries are not within the valid image area, the edge computing terminal generates a supplementary acquisition instruction containing a non-repeating random signature to uniquely identify the invalid acquisition event.

[0008] When processing a group of images from the same monitoring point, which includes leaf images and insect-attracting board images, the edge computing terminal pairs the leaf images and insect-attracting board images according to the acquisition time. When the leaf images and insect-attracting board images are within the same acquisition period, a group of field images is formed. When the leaf images and insect-attracting board images are not within the same acquisition period, a field image to be supplemented is formed, and the edge computing terminal generates a supplementary acquisition instruction.

[0009] The IoT gateway is used to upload suspected pest-infested areas and supplementary collection instructions to the server. Upon receiving a suspected pest-infested area, the IoT gateway writes the corresponding leaf image area, insect-attracting board image area, collection time, and monitoring point identifier into the same upload record, and then sends the upload record to the server. After receiving the upload record, the server establishes a pest identification record based on the monitoring point identifier and determines the correspondence between the leaf image area and the insect-attracting board image area in the pest identification record based on the collection time.

[0010] The server receives and parses supplementary acquisition commands. If multiple supplementary acquisition commands are received from the same monitoring point within a preset time window, the server triggers the adaptive parameter adjustment process of the semantic segmentation network on the edge computing terminal. The adaptive parameter adjustment process of the semantic segmentation network includes: the server generating loss function correction parameters that emphasize low-light robustness or resistance to motion ambiguity based on multiple consecutive invalid acquisition events, and sending them to the edge computing terminal via the IoT gateway to update the model parameters of the semantic segmentation network.

[0011] The server also associates leaf images and insect-attracting board images with the monitoring point identifier and collection time for received suspected pest-infested areas. Based on the pest sample database, historical pest records, and a spatial-temporal correlation analysis model, it correlates and fuses the distribution of insect spots in the leaf images with the insect count in the insect-attracting board images, and generates a pest warning level accordingly. The spatial-temporal correlation analysis model includes an attention mechanism module based on multi-source heterogeneous sensing data and a pest spread prediction module based on a graph neural network. The attention mechanism module dynamically weights the influence of the leaf spot area and the number of insects on the insect-attracting board, while the pest spread prediction module generates a pest area distribution prediction map for a future preset period based on current and historical correlation data.

[0012] When the server generates a pest warning level, it compares the image area of ​​the insect-attracting board in the uploaded record with the pest sample to form an insect identification level. It then determines the area and location of insect spots based on the leaf image area in the uploaded record to form a leaf damage level. Subsequently, a spatial-temporal correlation analysis model is used to correlate and fuse the insect identification level and the leaf damage level. When the insect identification level and the leaf damage level are consistent, the consistent level is used as the pest warning level; when they are inconsistent, the fused level output by the spatial-temporal correlation analysis model is used as the pest warning level.

[0013] The server is also used to send acquisition adjustment instructions, including adjustments to the acquisition frequency, to agricultural cameras based on the pest warning level, and to send warning instructions, including the pest warning level, to the warning execution equipment. After the server generates a pest warning level, it compares the pest warning level with a preset acquisition control level. When the pest warning level reaches the preset acquisition control level, an acquisition adjustment instruction is generated. The acquisition adjustment instruction includes the monitoring point identifier, acquisition time, and acquisition frequency. After receiving the acquisition adjustment instruction, the agricultural camera acquires leaf images and insect-attracting board images in the next acquisition cycle according to the monitoring point identifier and acquisition frequency in the acquisition adjustment instruction.

[0014] When the server generates a data collection adjustment command, it retrieves the pest warning levels generated by the same monitoring point within a continuous data collection period. If the pest warning level increases within the continuous data collection period, the data collection interval in the adjustment command is shortened; if the pest warning level does not increase within the continuous data collection period, the data collection interval in the adjustment command is maintained and written into the command. When the server sends a warning command to the warning execution device, it includes the pest warning level, monitoring point identifier, and data collection time. After receiving the warning command, the warning execution device determines the alarm method according to the pest warning level and executes the alarm action. After completing the alarm action, the device sends the warning execution information back to the server, which then writes the information into the pest identification record of the corresponding monitoring point identifier.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, leaf images and insect-attracting board images are collected from the same monitoring point using agricultural cameras. Before uploading, the edge computing terminal determines the validity of the images and extracts suspected pest areas. The server associates these two types of on-site images based on the monitoring point identifier and the collection time. Clarity and completeness conditions for the monitored object are used to distinguish between identified images and supplementary collection scenarios, reducing the number of invalid images entering the early warning determination. The server combines pest samples, historical pest records, and suspected pest areas to determine the pest early warning level and feeds back the collection adjustment instructions to the agricultural cameras. Upon receiving the early warning instructions, the early warning execution equipment executes the prompts according to the pest early warning level, ensuring a correspondence between the early warning output and subsequent collection. Attached Figure Description

[0016] Figure 1 This is the main flowchart for pest monitoring in this invention; Figure 2 This is a diagram illustrating the collaborative relationships within the system of this invention. Figure 3 This is a schematic diagram of image pairing for monitoring points in this invention; Figure 4 This is a valid state determination diagram for the edge-side image of the present invention; Figure 5 This is a schematic diagram of the space-time correlation analysis model of the present invention; Figure 6 This is a schematic diagram illustrating the data collection, feedback, and early warning execution of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0019] Please see Figures 1 to 6 This embodiment provides an intelligent monitoring system based on Internet of Things (IoT) machine vision. The system includes an agricultural camera, an edge computing terminal, an IoT gateway, a server, and an early warning execution device. The agricultural camera is deployed at positions corresponding to the crop leaves and insect-attracting boards. The edge computing terminal communicates with the agricultural camera, and the IoT gateway connects the edge computing terminal and the server. The server establishes an instruction interaction relationship with the early warning execution device and the agricultural camera. Within the agricultural monitoring area, the crop coverage area corresponding to a monitoring point is set to 20 to 80 square meters. The agricultural camera's acquisition resolution is set to 1920×1080 pixels, the distance from the agricultural camera lens to the crop canopy is set to 0.5 to 1.5 meters, and the effective area of ​​the insect-attracting board is set to 0.04 to 0.06 square meters. The aforementioned crop coverage area, acquisition resolution, lens distance, and effective area of ​​the insect-attracting board are jointly determined by the agricultural camera's field of view, the imaging proportion of the crop canopy in the image, and the proportion of the insect-attracting board in the image, to ensure that the leaf boundaries, insect-attracting board boundaries, insect-spotted areas, and insect-body areas form pixel areas in the on-site image that can be extracted by the edge computing terminal. After the agricultural camera acquires leaf images and insect-attracting board images at the same monitoring point, the edge computing terminal first performs semantic segmentation and effective status determination on the on-site images locally. The IoT gateway then uploads suspected pest areas or supplementary collection instructions to the server. The server generates pest warning levels based on image association, pest sample database, historical pest records, and spatial-temporal correlation analysis models, and controls the agricultural camera and warning execution equipment accordingly.

[0020] In this embodiment, the judgment data involving sharpness, boundary integrity, insect spot area, insect density, and fusion score are written into the server's configuration table through calibration records during system deployment. The calibration records include sharp field images, out-of-focus field images, motion-blurred field images, field images with missing leaf boundaries, field images with missing insect trap boundaries, insect spot labeled images, insect trap insect body labeled images, and historical insect infestation records. During subsequent operation, the server calls the judgment data in the configuration table to consistently process the upload records uploaded by the edge computing terminal.

[0021] Agricultural cameras collect leaf images and insect-attracting board images from the same monitoring point. Leaf images reflect the leaf boundary, internal leaf area, insect spot area, and insect spot distribution location. Insect-attracting board images reflect the board boundary, internal area, and insect count. When collecting images, the agricultural camera records its installation location, crop planting area, and insect-attracting board placement location as a monitoring point identifier, along with the collection time. The monitoring point identifier indicates that the leaf and insect-attracting board images originate from the same monitoring point, and the collection time is used to determine if they are within the same collection cycle. Under normal monitoring conditions, the agricultural camera collects images every 30 minutes. Once the pest warning level reaches the preset collection control level, the collection interval is adjusted to 15 minutes or 10 minutes according to the collection adjustment command. The collection time is recorded in the image format (year, month, day, hour, minute, second). The monitoring point identifier consists of the crop planting area number, the agricultural camera installation location number, and the insect-attracting board placement location number. The aforementioned 30-minute, 15-minute, and 10-minute intervals are written into the server's data collection frequency configuration table. The 30-minute interval is used as the regular monitoring interval, the 15-minute interval is used as the encrypted data collection interval under the medium pest warning level, and the 10-minute interval is used as the encrypted data collection interval under the high pest warning level.

[0022] Agricultural cameras send on-site images, including monitoring point identifiers and acquisition times, to an edge computing terminal. The on-site images include at least leaf images and insect-attracting board images; the monitoring point identifiers and acquisition times are sent to the edge computing terminal along with the on-site images. Upon receiving the on-site images, the edge computing terminal establishes image processing records using the monitoring point identifiers as indexes. On-site images with the same monitoring point identifiers are grouped into the same monitoring point image group; on-site images with different monitoring point identifiers have separate image processing records established. Leaf images and insect-attracting board images are compressed to at least 85% of their original image quality before transmission, with each on-site image having a data size of 300KB to 800KB. The aforementioned image quality and data size are determined by the compression output of the 1920×1080 pixel on-site images during edge-side transmission, ensuring that the on-site images retain pixel details of leaf boundaries, insect-attracting board boundaries, insect-marked areas, and insect-body areas, while simultaneously enabling the IoT gateway to transmit and upload the records. After receiving a group of images from the same monitoring point, the edge computing terminal uses the monitoring point identifier as the index field and the collection time as the time field, and writes the leaf image, the insect-attracting board image, the monitoring point identifier, and the collection time into the image processing record.

[0023] When an image group at the same monitoring point includes both leaf images and insect-attracting board images, the edge computing terminal pairs the leaf images and insect-attracting board images according to their acquisition time. If the leaf images and insect-attracting board images are within the same acquisition cycle, the edge computing terminal forms a group of on-site images, which then proceed to the subsequent image validity determination and suspected pest area extraction process. If the leaf images and insect-attracting board images are not within the same acquisition cycle, the edge computing terminal forms images to be supplemented and generates supplementary acquisition instructions. When pairing acquisition times, 60 seconds is used as the allowable deviation for the same acquisition cycle. This 60 seconds is determined by the continuous acquisition time difference between the agricultural camera and the insect-attracting board images, the receiving time difference of the edge computing terminal, and the transmission jitter of the IoT gateway, and must be less than the minimum encrypted acquisition interval of 10 minutes. If the difference between the acquisition time of the leaf image and the acquisition time of the insect-attracting board image does not exceed 60 seconds, the leaf images and insect-attracting board images are determined to be within the same acquisition cycle; if the difference exceeds 60 seconds, the leaf images and insect-attracting board images are determined to be outside the same acquisition cycle.

[0024] The edge computing terminal utilizes a pre-deployed lightweight semantic segmentation network to simultaneously process received leaf images and insect-attracting board images locally. The lightweight semantic segmentation network takes leaf and insect-attracting board images as input and outputs the leaf boundary, the insect-attracting board boundary, and the image regions within both boundaries. The leaf boundary determines the effective position of the crop leaf in the leaf image, the insect-attracting board boundary determines the effective position of the insect-attracting board in the insect-attracting board image, and the image regions are used to carry the subsequent extraction results of insect spot and insect body regions. Before receiving the leaf and insect-attracting board images, the lightweight semantic segmentation network uniformly scales them to 512×512 pixels. 512×512 pixels is the input size of the lightweight semantic segmentation network, and the local processing time for a single leaf or insect-attracting board image on the edge computing terminal is 0.2 to 0.8 seconds. This local processing time is determined by the edge computing terminal's processes of image scaling, semantic segmentation network inference, and pixel mask output. The leaf and insect-attracting board boundaries output by the lightweight semantic segmentation network are pixel masks, and the image region is formed by the set of pixels within the pixel mask.

[0025] The edge computing terminal determines the validity of an image based on the integrity of the leaf and insect-attracting board boundaries, as well as the sharpness of the extracted image regions. The determination of image validity includes both the leaf and insect-attracting board images. The edge computing terminal obtains the sharpness determination results for both the leaf and insect-attracting board images and verifies the leaf boundaries in the leaf image and the insect-attracting board boundaries in the insect-attracting board image. If the sharpness determination result reaches a preset sharpness threshold, and both the leaf and insect-attracting board boundaries are within the valid image area of ​​their respective images, the group of on-site images is determined to meet the image validity requirement. If the sharpness determination result does not reach the preset sharpness threshold, or if the leaf and insect-attracting board boundaries are not within the valid image area of ​​their respective images, the group of on-site images is determined not to meet the image validity requirement.

[0026] Sharpness determination uses the existing Laplace variance method, and the formula is: ; in, Rate the clarity. The grayscale image used for the determination. The response value of the grayscale image after processing with the Laplacian operator. This represents the variance of the Laplace response values. The preset sharpness threshold of 100 is determined by the sharpness calibration record during system deployment. In the sharpness calibration record, the sharpness of the on-site image... The lower limit is not less than 100 for out-of-focus and motion-blurred scene images. Since the upper limit is less than 100, 100 is written into the server's configuration table as the preset resolution threshold. When the value is not less than 100, the sharpness determination result reaches the preset sharpness threshold. When the value is less than 100, the sharpness determination result does not reach the preset sharpness threshold.

[0027] The integrity of leaf and insect-attracting board boundaries is determined by area coverage, using the following formula: ; in, For area coverage, This refers to the area inside the leaf boundary or insect-attracting board boundary extracted by a lightweight semantic segmentation network. This refers to the reference boundary area corresponding to the same monitoring point within the effective image area. Reference boundary area The coverage area is determined by clear field images obtained during system deployment. For the same monitoring point, the server writes the internal area of ​​the leaf boundary and the internal area of ​​the insect-attracting board boundary from the clear field image into the configuration table, respectively, as the reference boundary area for subsequent area coverage determination. 80% of the leaf boundary area is determined by the leaf boundary calibration records. In the leaf boundary calibration records, when the leaf boundary is obscured or missing by more than 20%, the insect spot area is likely to extend beyond the effective leaf area; therefore, 80% is written into the configuration table as the leaf boundary integrity threshold. 85% of the insect-attracting board boundary area is determined by the insect-attracting board boundary calibration records. The shape and area of ​​the insect-attracting board are relatively stable. When the insect-attracting board boundary is missing by more than 15%, the effective area of ​​the insect-attracting board corresponding to the insect count deviates; therefore, 85% is written into the configuration table as the insect-attracting board boundary integrity threshold. Not less than 80%, and the boundary of the insect-attracting board corresponds to When the percentage of leaf surface boundary and insect-attracting board boundary is not less than 85%, the leaf surface boundary and the insect-attracting board boundary are deemed to meet the integrity condition; the leaf surface boundary corresponds to Less than 80%, or the boundary of the insect trap corresponding to If the percentage is less than 85%, the leaf boundary and the insect-attracting board boundary are deemed not to meet the integrity requirements.

[0028] When the effective state of an image meets a preset sharpness threshold and both the leaf boundary and the insect-attracting board boundary are within the effective image area of ​​the corresponding image, the edge computing terminal extracts suspected pest areas containing insect spots or insect bodies from the image area based on the output of the semantic segmentation network. For leaf images, the edge computing terminal determines the insect spot area within the image area inside the leaf boundary and records the location range of the insect spot area in the leaf image. For insect-attracting board images, the edge computing terminal determines the insect body area within the image area inside the insect-attracting board boundary and records the location range of the insect body area in the insect-attracting board image. The edge computing terminal combines the insect spot area, the insect body area, the collection time, and the monitoring point identifier to form a suspected pest area. The minimum connected pixel area of ​​the insect spot area is 20 to 50 pixels, and the minimum connected pixel area of ​​the insect body area is 15 to 40 pixels. The aforementioned minimum connected pixel area is determined by the minimum effective imaging area of ​​the insect spot area and the insect body area in the 512×512 pixel image of the labeled sample; isolated areas smaller than the corresponding minimum connected pixel area are not included in the suspected pest area. The bounding boxes of the insect spot area and the insect body area record the pixel coordinates of the upper left corner, the pixel coordinates of the lower right corner, the area of ​​the region, and the image type to which it belongs. The image type to which it belongs includes leaf images and insect trap images.

[0029] When the image's valid state does not meet the preset clarity threshold, or when the leaf boundary or insect-attracting board boundary is not within the valid image range, the edge computing terminal generates a supplementary acquisition instruction containing a non-repeating random signature. This non-repeating random signature uniquely identifies the invalid acquisition event. The supplementary acquisition instruction retains the monitoring point identifier, acquisition time, and the non-repeating random signature. When generating the supplementary acquisition instruction, the edge computing terminal no longer uploads the corresponding on-site image as a suspected pest area; instead, it delegates the invalid acquisition event to the IoT gateway for uploading to the server. The non-repeating random signature is a random string of at least 16 characters, and it is not reused by the same edge computing terminal within the same acquisition date. The non-repeating random signature is generated by the edge computing terminal based on the monitoring point identifier, acquisition time, and a local random number, and written into the supplementary acquisition instruction for the server to exclude duplicate uploaded supplementary acquisition instructions. The supplementary acquisition instruction also includes an invalidity reason field, which includes insufficient clarity, incomplete leaf boundaries, incomplete insect-attracting board boundaries, and mismatched acquisition cycles.

[0030] The IoT gateway uploads suspected pest-infested areas and supplementary collection instructions to the server. Upon receiving a suspected pest-infested area, the IoT gateway writes the corresponding leaf image area, insect-attracting board image area, collection time, and monitoring point identifier into the same upload record and sends the record to the server. Upon receiving the upload record, the server establishes a pest identification record using the monitoring point identifier and determines the correspondence between the leaf image area and the insect-attracting board image area in the pest identification record using the collection time. Upon receiving a supplementary collection instruction, the IoT gateway sends the instruction to the server, enabling the server to identify invalid collection events based on the non-repeating random feature code in the supplementary collection instruction. The upload record includes the monitoring point identifier, collection time, leaf image area, insect-attracting board image area, area, bounding box coordinates, image validity status, and non-repeating random feature code. The data size of the same upload record ranges from 50KB to 300KB. This data size is determined by the local image area corresponding to the suspected pest-infested area, the bounding box coordinates, and necessary fields, and is less than the upload data size of a complete leaf image and a complete insect-attracting board image. After receiving the uploaded record, the server retrieves the pest identification record according to the monitoring point identifier; if an existing pest identification record is found, the uploaded record is written into the existing pest identification record; if no existing pest identification record is found, a new pest identification record is created.

[0031] The server receives and parses supplementary data collection instructions. Based on the monitoring point identifier, collection time, and non-repeating random feature code in the supplementary data collection instructions, the server counts supplementary data collection instructions corresponding to the same monitoring point within a preset time window. If the server receives multiple supplementary data collection instructions for the same monitoring point within the preset time window, the parameter adaptive adjustment process of the semantic segmentation network of the edge computing terminal is triggered. If the server does not receive multiple supplementary data collection instructions for the same monitoring point within the preset time window, the supplementary data collection instructions are only recorded as invalid data collection events and do not trigger the parameter adaptive adjustment process of the semantic segmentation network. The preset time window is 20 minutes. The 20 minutes is jointly determined by the minimum encrypted collection interval of 10 minutes and the number of consecutive invalid data collection events, ensuring that the same monitoring point can cover at least 3 collection opportunities under a high pest warning level. When the same monitoring point receives 3 or more supplementary data collection instructions within 20 minutes, it is determined that the same monitoring point has experienced multiple consecutive invalid data collection events; when the same monitoring point receives fewer than 3 supplementary data collection instructions within 20 minutes, the parameter adaptive adjustment process of the semantic segmentation network is not triggered. When the server collects supplementary collection instructions, it uses a non-repeating random signature to exclude duplicate uploads of the supplementary collection instructions.

[0032] The adaptive parameter adjustment process of the semantic segmentation network includes: the server generates loss function correction parameters based on multiple consecutive invalid acquisition events and distributes them to the edge computing terminal via the IoT gateway. When multiple consecutive invalid acquisition events correspond to low-light image states, the loss function correction parameters emphasize low-light robustness; when multiple consecutive invalid acquisition events correspond to motion-blurred image states, the loss function correction parameters emphasize anti-motion-blurring. After receiving the loss function correction parameters, the edge computing terminal uses them to update the model parameters of the semantic segmentation network. Low-light image states are marked as those with an average gray value of less than 60 in the field image. The average gray value of the field image is the arithmetic mean of the gray values ​​of the pixels involved in the judgment in the field image. The gray value of 60 is determined by the low-light calibration record. In the low-light calibration record, when the average gray value of the field image is less than 60, the continuity of the pixel mask edge of the leaf boundary or the insect-attracting board boundary decreases. Therefore, 60 is written into the configuration table as the marking threshold for low-light image states. Motion-blurred image states are scored according to sharpness. A score consistently below 100 is marked. The loss function correction parameter, used for weight adjustment during semantic segmentation network training or incremental updates, ranges from 0.1 to 0.5. When there are 3 to 4 consecutive invalid acquisition events, the correction parameter is 0.1; when there are 5 to 6 consecutive invalid acquisition events, it is 0.3; and when there are more than 6 consecutive invalid acquisition events, it is 0.5. When the invalidity reason is insufficient sharpness, the loss function correction parameter is used to adjust the participation weight of motion-blurred samples. When the invalidity reason is incomplete leaf or insect-attracting board boundaries and the average gray value of the on-site image is below 60, the loss function correction parameter is used to adjust the participation weight of low-light samples.

[0033] The server performs correlation and fusion on the received suspected pest areas. It associates leaf images and insect-attracting board images according to the monitoring point identifier and collection time, and, based on the pest sample database, historical pest records, and a spatial-temporal correlation analysis model, correlates and fuses the distribution of insect spots in the leaf images with the insect counts in the insect-attracting board images. The pest sample database provides a basis for insect comparison and pest type identification; historical pest records provide information on pest changes at the same monitoring point within a continuous collection period; and the spatial-temporal correlation analysis model establishes the correlation between insect spot distribution and insect counts between the same monitoring point and different collection times. Historical pest records retain the insect counts, insect spot area, pest warning level, and collection time for the same monitoring point within the most recent 24 hours. The 24-hour period is determined by the intraday variation cycle of agricultural pest records and is used to cover the changes in insect counts and insect spot area at the same monitoring point within a diurnal cycle. When the server performs correlation and fusion, it calls the historical insect infestation records of the most recent 4 to 8 collection cycles; when the historical insect infestation records of the same monitoring point are less than 4 collection cycles, the server calls all the historical insect infestation records that have been formed to participate in the correlation and fusion.

[0034] The spatial-temporal correlation analysis model includes an attention mechanism module based on multi-source heterogeneous sensing data and a pest spread prediction module built on a graph neural network. The attention mechanism module takes into account the area of ​​insect spots on leaves, the number of insects on the insect-attracting boards, and historical correlation data. It dynamically weights the influence of the leaf spot area and the number of insects on the insect-attracting boards and passes the weighted correlation results to the subsequent fusion process. The pest spread prediction module establishes the spread relationships between pest-affected areas based on current and historical correlation data, generating a pest area distribution prediction map for a preset future period. This pest area distribution prediction map, as one of the outputs of the spatial-temporal correlation analysis model, is used to determine the pest early warning level.

[0035] The degree of leaf damage is expressed using the existing formula for area percentage, which is: ; in, Leaf damage rate, The area of ​​the insect-spotted region in the leaf image area. This refers to the effective leaf surface area within the leaf boundary. The 2% and 8% damage levels are determined by the insect spot annotation images. In the insect spot annotation images, when the insect spot area occupies less than 2% of the effective leaf surface area, the insect spot area is scattered and is classified as low leaf damage level. When the insect spot area occupies not less than 2% and not more than 8% of the effective leaf surface area, the insect spot area forms a continuous or multi-point distribution in the leaf image and is classified as medium leaf damage level. When the insect spot area occupies more than 8% of the effective leaf surface area, the insect spot area forms a clearly expanded distribution in the leaf image and is classified as high leaf damage level. Less than 2% corresponds to a low level of leaf damage; When the damage rate is not less than 2% and not greater than 8%, it corresponds to the leaf damage level. A level greater than 8% corresponds to a high level of leaf damage.

[0036] The insect density on the insect trap is expressed using an existing density formula, which is: ; in, Insect density, This represents the number of insects identified within the image area of ​​the insect-attracting board. The effective area of ​​the insect-attracting board is 0.04 square meters to 0.06 square meters. The insect identification levels of 100 insects per square meter and 400 insects per square meter are determined by the effective area of ​​the insect-attracting board and the insect-labeled images on the board; when... When the density is between 0.04 and 0.06 square meters, 100 insects per square meter corresponds to approximately 4 to 6 insects per image on a single insect-attracting board, while 400 insects per square meter corresponds to approximately 16 to 24 insects per image on a single insect-attracting board. Below 100 insects per square meter, the insects in the insect-attracting board images are scattered; between 100 and 400 insects per square meter, the insects in the insect-attracting board images appear in a concentrated state; and above 400 insects per square meter, the insects in the insect-attracting board images reach a high density. When the number of insects per square meter is less than 100, it corresponds to a low insect identification level; When the number of insects per square meter is not less than 100 and not more than 400, the corresponding insect identification level is: A count greater than 400 insects per square meter corresponds to a high insect identification level.

[0037] When generating a pest warning level, the server compares the image area of ​​the insect-attracting board in the uploaded record with the pest sample to form an insect identification level. It then determines the area and location of insect spots based on the leaf image area in the uploaded record to form a leaf damage level. The insect identification level reflects the pest status corresponding to the insect type and count in the insect-attracting board image area, while the leaf damage level reflects the damage status corresponding to the area and location of insect spots in the leaf image area. The server then uses a spatial-temporal correlation analysis model to fuse the insect identification level and the leaf damage level. When the insect identification level and the leaf damage level are consistent, the consistent level is used as the pest warning level; when they are inconsistent, the fused level output by the spatial-temporal correlation analysis model is used as the pest warning level.

[0038] The space-time correlation analysis model uses the existing weighted summation method to form the fusion score, and the formula is as follows: ; in, For integrated scoring, This is the normalized value of insect body density. This represents the normalized value of leaf damage rate. This represents the normalized value for historical insect infestation changes. 0.45, 0.35, and 0.20 are the fusion weights in the server configuration table, and their sum is 1. The number of insects on the insect-attracting board is used to characterize the current insect activity status, therefore... The corresponding fusion weight was set to 0.45; the area of ​​insect spots on leaves was used to characterize the crop damage status, therefore The corresponding fusion weight is set to 0.35; historical insect infestation changes are used to correct the trend between the current collection cycle and the previous collection cycle, therefore The corresponding fusion weight is set to 0.20.

[0039] Based on insect density The mapping yields: When the number of heads is less than 100 per square meter, Take 0.2; When the number of heads per square meter is not less than 100 and not more than 400, Take 0.5; When the number of heads per square meter is greater than 400, Take 0.8. Leaf damage rate The mapping yields: When it is less than 2%, Take 0.2; When it is not less than 2% and not more than 8%, Take 0.5; When it is greater than 8%, Take 0.8. Mapped from the changes in pest warning levels within the most recent collection period: when the pest warning level did not increase within the most recent collection period. Take 0.2; when the pest warning level increases by one level in the most recent collection period, Take 0.5; when the pest warning level increases by two levels within the most recent collection period, Take 0.8. 0.2, 0.5, and 0.8 are written into the server configuration table as the center values ​​for low, medium, and high levels, respectively. 0.35 is the intermediate boundary value between 0.2 and 0.5, and 0.65 is the intermediate boundary value between 0.5 and 0.8.

[0040] When the value is less than 0.35, the spatial-temporal correlation analysis model outputs a low fusion level; The fusion level in the output of the spatial-temporal correlation analysis model is between 0.35 and 0.65; When the correlation coefficient is not less than 0.65, the spatial-temporal correlation analysis model outputs a high fusion level. When the insect identification level and the leaf damage level are inconsistent, the fusion level is used as the pest warning level.

[0041] The server sends a data acquisition adjustment command, which includes adjusting the acquisition frequency, to the agricultural camera based on the pest warning level, and also sends a warning command, which includes the pest warning level, to the warning execution device. After the server determines the pest warning level, it compares it with the preset acquisition control level. When the pest warning level reaches the preset acquisition control level, the server generates an acquisition adjustment command. This command includes the monitoring point identifier, acquisition time, and acquisition frequency. Upon receiving the command, the agricultural camera acquires leaf and insect-attracting board images in the next acquisition cycle according to the monitoring point identifier and acquisition frequency specified in the command. If the pest warning level does not reach the preset acquisition control level, the agricultural camera continues acquisition at the original acquisition frequency. The preset acquisition control level is a medium pest warning level. When the pest warning level is medium or high, the server generates an acquisition adjustment command; when the pest warning level is low, the server does not generate an acquisition adjustment command, and the agricultural camera maintains a acquisition cycle of once every 30 minutes. The preset acquisition control level is determined by the server configuration table and is used to distinguish between routine and intensive monitoring acquisition.

[0042] When the server generates a data collection adjustment command, it retrieves the pest warning levels generated at the same monitoring point within a continuous data collection cycle. If the pest warning level increases within a continuous data collection cycle, the server shortens the data collection interval in the adjustment command; if the pest warning level does not increase within a continuous data collection cycle, the server maintains the data collection interval in the adjustment command and writes it into the command. After receiving the adjustment command, the agricultural camera executes the acquisition of leaf images and insect-attracting board images for the next data collection cycle according to the monitoring point identifier corresponding to the command. The continuous data collection cycle is the most recent three cycles. These three cycles are used to determine whether the pest warning level has changed continuously, avoiding adjustments to the data collection interval triggered by a single fluctuation in the field image. If the pest warning level changes from low to medium within the most recent three cycles, the data collection interval is shortened from 30 minutes to 15 minutes; if it changes from medium to high, the interval is shortened from 15 minutes to 10 minutes; if the pest warning level does not increase within the most recent three cycles, the interval remains unchanged.

[0043] When the server sends an early warning command to the early warning execution device, it includes the pest early warning level, monitoring point identifier, and collection time in the command. Upon receiving the command, the early warning execution device determines the alarm method according to the pest early warning level and executes the alarm action. After completing the alarm action, the device sends early warning execution information back to the server, which then writes this information into the pest identification record corresponding to the monitoring point identifier. The pest identification record includes the upload record, pest identification level, leaf damage level, pest early warning level, collection adjustment command, and early warning execution information, creating a closed-loop record for the same monitoring point, from on-site image acquisition, edge-side effective status determination, server-side association and fusion to early warning execution. Low pest early warning levels correspond to a display prompt on the early warning execution device; medium levels correspond to intermittent audible and visual alarms; and high levels correspond to continuous audible and visual alarms. The early warning execution information includes at least the early warning command reception time, alarm method, alarm action completion time, and corresponding monitoring point identifier. The aforementioned alarm method is determined by the server's pest early warning level configuration table and sent to the early warning execution device along with the early warning command.

[0044] In this embodiment, an agricultural camera, an edge computing terminal, an IoT gateway, a server, and an early warning execution device constitute a collaborative link for on-site data acquisition, edge processing, IoT transmission, server-side association and fusion, and on-site early warning execution. Leaf images and insect-attracting board images are paired within the same monitoring point and the same acquisition cycle. Suspected pest areas are entered into the server-side pest identification record based on the acquisition time and monitoring point identifier. A semantic segmentation network is used to output leaf boundaries, insect-attracting board boundaries, and image regions at the edge side. A spatial-temporal correlation analysis model is used to associate and fuse insect spot distribution, insect count, and historical pest records at the server side. Non-repeating random feature codes in supplementary acquisition instructions are used to identify invalid acquisition events. Acquisition adjustment instructions are used to align the subsequent acquisition frequency of the agricultural camera with the pest early warning level. Early warning instructions are used to instruct the early warning execution device to execute alarm actions according to the pest early warning level. According to the above data settings, a monitoring point generates 2 sets of on-site images per hour under normal monitoring conditions, 4 sets of on-site images per hour under medium pest warning level, and 6 sets of on-site images per hour under high pest warning level. After the edge computing terminal extracts the suspected pest area, the server writes it into the continuous pest identification record with the monitoring point identifier, collection time, and pest warning level.

[0045] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An intelligent monitoring system based on Internet of Things (IoT) machine vision, characterized in that, This includes agricultural cameras, edge computing terminals, IoT gateways, servers, and early warning execution devices; The agricultural camera is used to collect leaf images and insect-attracting board images at the same monitoring point, and sends the on-site images with monitoring point identification and collection time to the edge computing terminal; The edge computing terminal is used for: Using a pre-deployed lightweight semantic segmentation network, the received leaf images and insect-attracting board images are processed locally in a synchronized manner to extract the leaf boundaries, insect-attracting board boundaries, and image regions within them. The validity of the image is determined based on the integrity of the leaf surface boundary and the insect-attracting board boundary, as well as the clarity of the extracted image area. When the effective state of the image meets the preset clarity threshold and the leaf boundary and the insect-attracting board boundary are both within the effective screen area of ​​the corresponding image, based on the output result of the semantic segmentation network, the suspected pest area containing insect spots or insects is extracted from the image area. When the valid state of the image does not meet the preset sharpness threshold or the leaf boundary or insect-attracting board boundary is not within the valid image range, a supplementary acquisition instruction containing a non-repeating random feature code is generated to uniquely identify the invalid acquisition event. The IoT gateway is used to upload the suspected pest-infested area and the supplementary collection instruction to the server. The server is used for: The system receives and parses the supplementary acquisition instructions. If multiple supplementary acquisition instructions for the same monitoring point are received within a preset time window, the system triggers the parameter adaptive adjustment process of the semantic segmentation network of the edge computing terminal. For the received suspected pest areas, the leaf images and insect-attracting board images are associated according to the monitoring point identification and the collection time. Based on the pest sample database, historical pest records, and using a spatial-temporal correlation analysis model, the distribution of insect spots in the leaf images and the insect count in the insect-attracting board images are correlated and fused, and a pest warning level is generated accordingly. The server is also used to send a data acquisition adjustment instruction, which includes adjusting the acquisition frequency, to the agricultural camera according to the pest warning level, and to send a warning instruction, which includes the pest warning level, to the warning execution device.

2. The intelligent monitoring system based on IoT machine vision according to claim 1, characterized in that, The space-time correlation analysis model includes: An attention mechanism module based on multi-source heterogeneous sensing data is used to dynamically weight the influence of leaf spot area and insect number on the insect trap; and The pest spread prediction module, built on graph neural networks, is used to generate a predicted map of pest regional distribution for a future preset period based on current and historical correlation data.

3. The intelligent monitoring system based on IoT machine vision according to claim 1, characterized in that, The adaptive parameter adjustment process of the semantic segmentation network includes: Based on multiple consecutive invalid acquisition events, the server generates a loss function correction parameter that emphasizes low-light robustness or resistance to motion blur, and sends it to the edge computing terminal via the IoT gateway to update the model parameters of the semantic segmentation network.

4. The intelligent monitoring system based on Internet of Things machine vision according to claim 1, characterized in that, When the agricultural camera collects leaf images and insect-attracting board images at the same monitoring point, the installation location of the agricultural camera, the crop planting area, and the placement location of the insect-attracting board are written into the monitoring point identifier of the collected images. After receiving the acquired image, the edge computing terminal establishes an image processing record using the monitoring point identifier as an index; Images acquired with the same monitoring point identifier are grouped into the same monitoring point image group, while images acquired with different monitoring point identifiers are recorded separately.

5. The intelligent monitoring system based on IoT machine vision according to claim 4, characterized in that, The image group at the same monitoring point includes leaf images and insect-attracting board images. The edge computing terminal pairs the leaf images and the insect-attracting board images according to the acquisition time. When the leaf images and the insect-attracting board images are within the same acquisition period, a set of field images is formed; When the leaf image and the insect-attracting board image are not in the same acquisition cycle, a supplementary on-site image is generated, and the edge computing terminal generates a supplementary acquisition command.

6. The intelligent monitoring system based on Internet of Things machine vision according to claim 1, characterized in that, After receiving the suspected pest area, the IoT gateway writes the leaf image area, insect-attracting board image area, collection time, and monitoring point identifier corresponding to the suspected pest area into the same upload record, and sends the upload record to the server. After receiving the uploaded records, the server establishes pest identification records based on the monitoring point identifiers and determines the correspondence between the leaf image area and the insect-attracting board image area in the pest identification records based on the collection time.

7. The intelligent monitoring system based on IoT machine vision according to claim 6, characterized in that, When the server generates a pest warning level, it compares the insect-attracting board image area in the uploaded record with the pest sample to form an insect identification level, and determines the insect spot area and insect spot distribution location based on the leaf image area in the uploaded record to form a leaf damage level. Then, the spatial-temporal correlation analysis model is used to correlate and fuse the insect identification level and the leaf damage level. When the insect identification level and the leaf damage level are consistent, the consistency level is used as the pest warning level. When the insect identification level is inconsistent with the leaf damage level, the fusion level output by the spatial-temporal correlation analysis model shall be used as the pest warning level.

8. The intelligent monitoring system based on Internet of Things machine vision according to claim 1, characterized in that, After the server generates the pest warning level, it compares the pest warning level with the preset collection and control level. When the pest warning level reaches the preset collection control level, a collection adjustment command is generated; The data collection and adjustment instructions are written into the monitoring point identifier, collection time, and collection frequency; After receiving the acquisition and adjustment instruction, the agricultural camera will acquire leaf images and insect-attracting board images in the next acquisition cycle according to the monitoring point identification and acquisition frequency in the acquisition and adjustment instruction.

9. The intelligent monitoring system based on IoT machine vision according to claim 8, characterized in that, When the server generates the aforementioned data collection and adjustment instruction, it retrieves the pest early warning level generated by the same monitoring point within a continuous data collection period. When the pest warning level rises during a continuous collection period, shorten the collection interval in the collection adjustment instruction. If the pest warning level does not rise within the continuous collection period, maintain the collection interval in the collection adjustment instruction and write the collection interval into the collection adjustment instruction.

10. The intelligent monitoring system based on Internet of Things machine vision according to claim 1, characterized in that, When the server sends an early warning command to the early warning execution device, it writes the pest early warning level, monitoring point identifier, and collection time into the early warning command. After receiving the early warning instruction, the early warning execution device determines the alarm method and executes the alarm action according to the pest early warning level. After the early warning execution device completes the alarm action, it sends the early warning execution information back to the server, and the server writes the early warning execution information into the pest identification record of the corresponding monitoring point.

Images