An intelligent breeding inspection method and system based on cloud-edge collaboration and multi-modal spatial activity analysis

Abstract

This invention belongs to the interdisciplinary field of smart agriculture and intelligent robotics, specifically relating to an intelligent livestock inspection method and system based on cloud-edge collaboration and multimodal spatial activity analysis. It includes a cloud platform, an edge computing terminal, a hardware terminal, and a real-time control terminal. The cloud platform sends inspection tasks to the edge computing terminal. The edge computing terminal parses the inspection tasks and configures the corresponding operating mode according to the task type. The hardware terminal collects laser scanning data, video data, environmental parameters, posture data, ultrasonic distance, and wheel speed-mileage data from the livestock farm and transmits the data to the edge computing terminal and the real-time control terminal. This invention solves the problems in existing livestock inspection technologies, such as unclear boundaries between high-computing-power tasks and real-time control tasks leading to control cycle jitter, lack of a unique selection mechanism for multiple control sources leading to command conflicts, and inconsistent spatiotemporal references between animal trajectories, environmental parameters, and robot poses leading to data fragmentation.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of smart agriculture and intelligent robots, specifically involving an intelligent aquaculture inspection method and system based on cloud-edge collaboration and multimodal spatial activity analysis. Background Technology

[0002] With the rapid development of modern livestock and poultry farming towards large-scale and intensive operations, the requirements for environmental monitoring, animal health management, and production automation are constantly increasing. Traditional manual inspection methods are time-consuming, labor-intensive, prone to missed or incorrect inspections, and have slow response times, making them insufficient to meet the needs of modern refined livestock management. Robotic inspection, with its advantages of all-weather operation, high precision, and high efficiency, is considered an important vehicle for connecting environmental perception, animal status observation, and production management.

[0003] Existing livestock inspection technologies are mainly divided into three categories: fixed facility sensing, track-based mobile inspection, and autonomous mobile robots. Fixed facilities are limited by installation location and obstructions, and cannot actively cover blind spots; track-based equipment requires modification of the pen infrastructure, and lacks path flexibility and temporary obstacle avoidance capabilities; although autonomous mobile robots can actively expand the coverage area, they still have the following shortcomings in the complex environment of livestock farms:

[0004] First, the boundary between high-computing-power tasks and real-time control tasks is unclear. Tasks such as synchronous positioning and mapping, and visual reasoning have unstable computational loads, while motor closed-loop control and emergency stop processing require stable control cycles. Running both on the same processor can easily cause cycle jitter, affecting the real-time performance and safety of chassis control.

[0005] Secondly, there is a lack of hierarchical arbitration mechanisms between various upper-level motion control sources and local safety controls. Autonomous navigation, visual line following, target tracking, and remote manual control may all generate upper-level speed control quantities. Without a unique control source selection mechanism, command conflicts are likely to occur. At the same time, safety events such as physical emergency stops, local controller takeover, control communication timeouts, and chassis failures need to be handled with higher priority at the real-time control end. Existing solutions still lack a unified design for the boundaries of responsibility and priority relationships between upper-level motion control and lower-level safety coverage.

[0006] Third, animal trajectories, environmental parameters, and robot poses are disconnected. Visual output, environmental data, and map coordinates use their own independent temporal and spatial references, making it difficult to form a unified, traceable record and effectively support comprehensive queries by individual, region, and time period.

[0007] Fourth, data continuity and security strategies are insufficient under weak network conditions. Livestock farm networks experience frequent fluctuations, and after network outages, structured data lacks local caching, sequence number verification, and recovery / retransmission mechanisms, easily leading to data loss or duplication.

[0008] Therefore, it is necessary to invent an intelligent aquaculture inspection method and system based on cloud-edge collaboration and multimodal spatial activity analysis. Summary of the Invention

[0009] To address the problems in existing aquaculture inspection technologies, such as unclear boundaries between high-computing tasks and real-time control tasks leading to control cycle jitter, lack of a unique selection mechanism for multiple control sources causing command conflicts, inconsistent spatiotemporal references between animal trajectories, environmental parameters, and robot poses leading to data fragmentation, and lack of local caching and recovery / retransmission mechanisms under weak network conditions resulting in data loss or duplication, this invention provides an intelligent aquaculture inspection method and system based on cloud-edge collaboration and multimodal spatial activity analysis.

[0010] This invention is achieved using the following technical solution:

[0011] A smart aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis includes:

[0012] It provides cloud, edge computing, hardware, and real-time control terminals;

[0013] The cloud sends inspection tasks to the edge computing terminal. The inspection tasks include task identifier, inspection area, target point, operation mode and security parameters.

[0014] The edge computing terminal parses the inspection task and configures the corresponding running mode according to the task type;

[0015] The hardware terminal collects laser scanning data, video data, environmental parameters, posture data, ultrasonic distance, and wheel speed mileage data within the farm, and transmits the laser scanning data and video data to the edge computing terminal, while transmitting the environmental parameters, posture data, ultrasonic distance, and wheel speed mileage data to the real-time control terminal.

[0016] The edge computing terminal performs synchronous positioning and mapping based on laser scanning data and wheel speed and mileage data to obtain robot pose and map, and generates navigation speed control values ​​based on robot pose, target point and local obstacles in laser scanning data;

[0017] The real-time control terminal performs safety coverage on the navigation speed control quantity, and then performs differential chassis kinematics calculation and wheel speed closed-loop adjustment;

[0018] The edge computing terminal performs instance segmentation, multi-target tracking, and trajectory clustering on video data to obtain continuous trajectories and activity hotspots with behavior categories and trajectory identifiers;

[0019] The edge computing terminal associates environmental parameters, robot pose, device status, behavior category, continuous trajectory, and activity hotspots fed back from the real-time control terminal into structured inspection records;

[0020] When the network is normal, structured inspection records are uploaded to the cloud in real time; when the network is interrupted, unconfirmed records are written to the local cache; after the network is restored, the records are re-uploaded according to their sequence numbers and deduplicated by the cloud based on the unique identifier of each record.

[0021] Furthermore, when the hardware acquires data, it transmits laser scanning data to the edge computing terminal via a two-dimensional LiDAR and video data via a front-facing camera and an infrared PTZ camera. The real-time control terminal reads environmental parameters from the environmental transmitter via an RS485 interface, including temperature, humidity, ammonia, hydrogen sulfide, and carbon dioxide data. It also reads attitude data via an SPI interface, outputs a trigger signal via GPIO, measures the echo pulse width to obtain ultrasonic distance, and acquires wheel speed and mileage data via a timer encoder interface. The real-time control terminal also generates PS2 handle takeover status, physical emergency stop status, and fault status as discrete status data, and encapsulates the discrete status data along with environmental parameters, attitude data, ultrasonic distance, and wheel speed and mileage data for transmission to the edge computing terminal.

[0022] Furthermore, the operating modes include autonomous navigation, visual line following, target following, and remote manual control. The edge computing terminal selects the only valid upper-level control source from among the four candidate control sources (autonomous navigation, visual line following, target following, and remote manual control) based on the current operating mode. It then applies amplitude and acceleration limits to the selected linear and angular velocities, performs control timeliness verification, and sends the arbitrated linear and angular velocities to the real-time control terminal via USB CDC. The real-time control terminal performs safety processing on the linear and angular velocities sent from the edge computing terminal, including physical emergency stop processing, PS2 local takeover processing, control communication timeout processing, and chassis fault processing. It then calculates the target angular velocities of the left and right wheels according to the two-wheel differential chassis kinematic model and uses incremental PID control based on encoder feedback to perform closed-loop wheel speed adjustment.

[0023] Furthermore, the edge computing terminal performs instance segmentation, multi-target tracking, and trajectory clustering on video data, specifically including:

[0024] Infrared video frames are input into the instance segmentation model, and the target bounding box, instance mask, behavior category and model confidence are output frame by frame.

[0025] Predict the current target position based on the target state of the previous moment, and maintain the trajectory identifier based on spatial overlap and matching cost. Output the tracking result, which includes at least the trajectory identifier, timestamp, center point coordinates and matching state.

[0026] Based on the camera calibration parameters, robot pose, and enclosure coordinate system, the pixel coordinates of the tracking center point are converted into the actual planar coordinates of the enclosure.

[0027] The trajectory points are clustered by density according to the set time window, neighborhood radius and minimum number of samples, high-density dwell areas are identified and outliers are removed, and individual dwell areas or group activity hotspots are output.

[0028] Furthermore, the structured inspection record includes at least the task identifier, record unique identifier, equipment identifier, enclosure identifier, edge timestamp, robot pose, coordinate system name, environmental parameters, wheel speed and mileage, trajectory points, cluster number, trajectory identifier, behavior category, behavior category confidence level, bounding box, tracking status, data quality marker, algorithm version, and sequence number. When generating the structured inspection record, if the sampling frequencies of data from different sources are inconsistent, the edge computing terminal will use the nearest time matching, time window summarization, or interpolation method according to the record generation time to complete the alignment, and record the matching time difference and data quality status.

[0029] Furthermore, when environmental parameters exceed preset thresholds, or when an individual's activity level, dwell time, and activity area continuously exceed preset deviation thresholds relative to a preset baseline within a preset time window, the edge computing terminal generates an abnormal event record containing the abnormality type, occurrence time, enclosure location, associated trajectory identifier, trigger data, and associated video clip index.

[0030] Furthermore, the structured inspection records are uploaded in real time by the edge computing terminal via MQTT or HTTP, while the video data is transmitted independently via RTSP or RTMP, ensuring that the video stream and structured data belong to different links. When the network is interrupted, the edge computing terminal writes the unconfirmed records into the local cache according to the task identifier, record unique identifier, sequence number, and data priority. After the network is restored, the edge computing terminal retransmits the unconfirmed records according to the sequence number, and the cloud performs deduplication on the records already in the database based on the record unique identifier.

[0031] An intelligent aquaculture inspection system based on cloud-edge collaboration and multimodal spatial activity analysis includes:

[0032] The cloud-based interaction and management module, deployed on a cloud server, is used to issue inspection tasks, receive structured inspection data and write it into the database, and perform deduplication on supplementary records based on the unique identifier of the record.

[0033] The edge computing and motion control module, deployed on edge computing devices, is used to analyze inspection tasks, complete synchronous localization and mapping to obtain robot pose and map, path planning, visual line following, target following, motion command arbitration, and associate environmental parameters, robot pose, device status, behavior category, continuous trajectory and activity hotspots into structured inspection records based on time and space correlations.

[0034] The real-time control module, deployed on the real-time controller, has logically independent uplink data channels and downlink control channels with the edge computing device. It is used to collect environmental parameters, attitude data, ultrasonic distance, wheel speed and mileage data and device status, perform safety coverage on the linear velocity and angular velocity sent by the edge computing device, and complete differential chassis kinematics calculation and wheel speed closed-loop adjustment.

[0035] The hardware sensing and execution module includes a 2D LiDAR, a front-facing camera, an infrared PTZ camera, an environmental transmitter, an attitude sensor, an ultrasonic module, an encoder, a motor driver, and a motor. It is used to acquire laser scanning data, video data, environmental parameters, attitude data, ultrasonic distance, and wheel speed and mileage data, and to execute motion commands.

[0036] Furthermore, the edge computing and motion control module includes a motion command arbitration unit, which receives four types of candidate control sources: autonomous navigation, visual line following, target following, and remote manual control. Based on the current operating mode, it selects the only valid upper-level control source, applies amplitude and acceleration limits to the selected linear and angular velocities, performs control timeliness verification, and sends the arbitration-resolved linear and angular velocities to the real-time control module. The real-time control module includes a local control arbitration and emergency stop safety processing unit, which performs safety processing on the linear and angular velocities sent by the edge computing device, including physical emergency stop processing, PS2 local takeover processing, control communication timeout processing, and chassis fault processing. It also calculates the target angular velocities of the left and right wheels according to the two-wheel differential chassis kinematic model and uses incremental PID control based on encoder feedback to perform closed-loop wheel speed adjustment.

[0037] Furthermore, the edge computing and motion control module also includes a structured data communication and network outage retransmission unit, which is used to upload structured inspection records in real time when the network is normal, and write unconfirmed records to the local cache according to task identifier, record unique identifier, sequence number and data priority when the network is interrupted. After the network is restored, the unconfirmed records are retransmitted according to the sequence number, and the cloud performs deduplication on the records already in the database according to the record unique identifier.

[0038] Compared with existing technologies, this invention provides an intelligent aquaculture inspection method and system based on cloud-edge collaboration and multimodal spatial activity analysis, which has the following beneficial effects:

[0039] 1. This invention employs a layered collaborative processing approach involving the cloud, edge computing, real-time control, and hardware. The edge computing unit handles high-computing tasks such as synchronous localization and mapping, path planning, instance segmentation, and trajectory analysis, while the real-time control unit handles low-latency real-time control tasks such as emergency stop processing, kinematic calculation, wheel speed closed-loop adjustment, and execution device control. This resolves the contradiction between high-computing algorithms and low-latency real-time control during centralized operation, enabling the robot to maintain the stability of its chassis control cycle while performing vision and navigation calculations.

[0040] 2. This invention employs a two-level control mechanism combining Jetson Nano upper-level motion command arbitration with STM32F407 local safety coverage. Jetson Nano selects a single valid upper-level control source from four candidate control sources: autonomous navigation, visual line following, target following, and remote manual control. STM32F407 then performs safety coverage such as physical emergency stop processing, PS2 local takeover processing, control communication timeout processing, and chassis fault handling. This solves the problem of command conflicts caused by the lack of a unique selection mechanism when multiple control sources coexist, thus improving the system's safety and reliability.

[0041] 3. This invention obtains continuous trajectories with behavior labels through behavior category recognition, instance segmentation, and continuous tracking. After converting the trajectory points to the actual spatial coordinates of the enclosure by combining camera calibration parameters and robot pose, density clustering is performed to obtain individual or group activity hotspots. Environmental parameters, robot pose, equipment status, behavior category, continuous trajectory, and activity hotspots are aligned in time and spatially according to a unified record structure to form a traceable structured inspection record. This solves the problem of data fragmentation caused by the inconsistency of spatiotemporal references between animal trajectory, environmental parameters, and robot pose, enabling managers to perform comprehensive queries and analyses based on trajectory identifiers, regions, time periods, and inspection tasks.

[0042] 4. This invention employs a complete communication mechanism that records unique identifiers, data sequence numbers, local caches, confirmation status, recovery and retransmission, and cloud deduplication. When the network is normal, structured records are uploaded in real time. When the network is interrupted, unconfirmed records are written to the local cache. After the network is restored, the records are retransmitted according to the sequence number, and the cloud performs deduplication based on the unique identifier of the record. This solves the problems of easy loss of inspection data when the network fluctuates or is interrupted in the farm and easy rewriting after the upload is restored. This ensures that the structured inspection data remains continuous, queryable, and traceable. Attached Figure Description

[0043] Figure 1 This is a diagram illustrating the system architecture and end-to-end data interaction of the present invention.

[0044] Figure 2 This is a flowchart of the method of the present invention.

[0045] Figure 3 This is a diagram illustrating the dual-core collaborative control and data closed-loop of the present invention.

[0046] Figure 4 This is the motion mode state transition diagram based on the finite state machine of the present invention. Detailed Implementation

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0048] This invention provides an intelligent aquaculture inspection method and system based on cloud-edge collaboration and multimodal spatial activity analysis. For example... Figure 1 As shown, the system adopts a four-layer overall architecture, including a cloud interaction and management layer, an edge computing and motion control layer, a real-time control layer, and a hardware perception and execution layer. The edge computing and motion control layer is deployed on a Jetson Nano edge computing device. This layer uses the ROS robot operating system to realize communication and scheduling between various functional nodes within the edge computing device, communicates with the cloud via MQTT or HTTP, and performs bidirectional data interaction with the real-time controller via USB CDC. The real-time control layer is deployed on an STM32F407 real-time controller. The hardware perception and execution layer includes various sensors and actuators. The four layers correspond to the cloud, edge computing, real-time control, and hardware ends, respectively.

[0049] The cloud-based interaction and management layer (cloud server) is responsible for task management, data services, historical queries, anomaly alerts, and business display. Administrators set inspection areas, target points, operating modes, environmental thresholds, and anomaly conditions via WeChat mini-programs or a web management interface. The cloud then sends tasks to the edge communication gateway. This layer is used to issue inspection tasks, receive structured inspection data and write it to the database, and perform deduplication on retransmitted records based on unique record identifiers. Backend services save environment, pose, trajectory, hotspots, and anomaly records according to task identifiers, device identifiers, record identifiers, and sampling times. Infrared video streams do not pass through the structured data gateway; instead, they are uploaded by Jetson video access and forwarding nodes to the cloud RTSP access or RTMP forwarding service, and then played on the real-time monitoring page.

[0050] The edge computing and motion control layer is responsible for task parsing, synchronous localization and mapping, path planning, visual line following, target following, animal behavior category recognition and instance segmentation, continuous trajectory tracking, trajectory coordinate transformation, spatial activity analysis, motion command arbitration, and multimodal data association. This layer includes an MQTT or HTTP communication gateway, task parsing and mode configuration, 2D LiDAR data access, synchronous localization and mapping, path planning and navigation, RGB video access, HSV visual line following, KCF target following, remote manual control, motion command arbitration, structured data communication and network outage retransmission unit, and Jetson ROS communication and USB CDC node. Among them, the motion command arbitration unit is used to receive four types of candidate control sources: autonomous navigation, visual line following, target following, and remote manual control. Based on the current operating mode, it selects the only valid upper-layer control source, applies amplitude and acceleration limits to the selected linear and angular velocities, performs control timeliness verification, and sends the arbitrated linear and angular velocities to the real-time control layer. The structured data communication and network outage retransmission unit is used to upload structured inspection records in real time when the network is normal. When the network is interrupted, unconfirmed records are written to the local cache according to the task identifier, record unique identifier, sequence number and data priority. After the network is restored, the unconfirmed records are retransmitted according to the sequence number, and the cloud performs deduplication on the records already in the database according to the record unique identifier.

[0051] The real-time control layer (STM32F407 real-time controller) is responsible for sensor data acquisition, emergency stop processing, chassis kinematics calculation, wheel speed closed-loop regulation, and actuator output. For example... Figure 1 and Figure 3 As shown, this layer has logically independent uplink data channels and downlink control channels with the edge computing device. These channels are used to collect environmental parameters, attitude data, ultrasonic distance, wheel speed and mileage data, and device status. They also perform safety overlay on the linear and angular velocities sent by the edge computing device, and complete differential chassis kinematics calculations and wheel speed closed-loop adjustment. Figure 3As shown, this layer includes a local control arbitration and emergency stop safety processing unit. This unit performs safety processing on the linear and angular velocities sent by the edge computing device, including physical emergency stop processing, PS2 local takeover processing, control communication timeout processing, and chassis fault processing. It also calculates the target angular velocities of the left and right wheels according to the two-wheel differential chassis kinematic model and uses incremental PID control based on encoder feedback for closed-loop wheel speed adjustment. The real-time control layer sets up logically independent uplink acquisition chains and downlink control chains. The uplink acquisition chain reads environmental, attitude, ultrasonic, and encoder data, performing basic filtering, status judgment, unified packaging, and USB CDC uplink encapsulation. The PS2 controller takeover status, physical emergency stop status, current local control source, and fault codes are generated as discrete status information by the local control arbitration and emergency stop safety processing unit and encapsulated and uploaded together with sensor data during the data packaging stage. The downlink control chain parses the only valid upper-layer control quantity and audio-visual control commands sent by the Jetson Nano. The local security overlay mechanism works as follows: The STM32F407 checks the frame header, length, sequence number, checksum, and control timeliness of the control frames sent by the Jetson Nano. If no valid control frame is received for more than the set control cycle, the control communication is deemed to have timed out, the original downlink control quantity is blocked, and zero speed is output. When an emergency stop input, a serious chassis fault, an encoder malfunction, or the proximity safety limit is reached, the motor output is shut down or limited according to the corresponding safety level.

[0052] The hardware sensing and execution layer includes a 2D LiDAR, a front-facing camera, an infrared PTZ camera, an environmental transmitter, an attitude sensor (ICM-20948), an ultrasonic module (HC-SR04), an encoder, a motor driver, left / right drive motors, an RGB LED strip, and a buzzer. It is responsible for acquiring environmental, video, attitude, distance, and wheel speed information, as well as executing motion commands. The 2D LiDAR, front-facing camera, and infrared PTZ camera directly provide scan data, RGB video frames, and RTSP video streams to the Jetson Nano. The environmental transmitter, attitude sensor, ultrasonic module, PS2 wireless receiver, physical emergency stop switch, and encoder are connected to the STM32F407. PWM, direction, and enable signals enter the motor driver and drive the left and right motors respectively. GPIO or PWM signals control the RGB LED strip and buzzer respectively.

[0053] The connection relationships and interface definitions between the layers of the system and between the layers and the hardware are shown in Table 1.

[0054] Table 1. Main Connections and Interface Definitions of the System

[0055]

[0056] like Figure 2As shown, the intelligent aquaculture inspection method is executed in the following order: task reception and parsing, data collection, positioning, navigation and motion control, animal spatial activity analysis, multimodal association, cloud uploading and anomaly recovery. Each step is uniformly numbered from S101 to S110.

[0057] S101: Task Creation and Issuance

[0058] The cloud-based interaction and management layer receives inspection areas, target points, operating modes, algorithm parameters, environmental thresholds, query conditions, and stop requests submitted via WeChat Mini Program or Web management terminal. The management terminal organizes the inspection tasks into structured messages containing task identifier, robot identifier, target point set, operating mode, safe speed, environmental threshold, task validity period, and stop fields. After backend service performs field validation, task number verification, and permission verification, it sends the message to the JetsonNano communication gateway via MQTT or HTTP.

[0059] S102: Task Analysis and Mode Configuration

[0060] The edge communication gateway parses the task field, and the Jetson Nano configures standby, autonomous navigation, visual line following, target following, or remote manual control states based on the task type and device status. The operating modes include autonomous navigation, visual line following, target following, and remote manual control. HSV visual line following and KCF target following are independent operating modes and do not constitute a serial processing relationship. PS2 controller takeover, safety stop, and emergency stop are triggered by the STM32F407 based on local input and safety events.

[0061] S103: Multi-source data acquisition

[0062] like Figure 1 and Figure 3 As shown, the multi-source data acquisition process provides a unified data entry point for robot localization, motion control, animal spatial activity analysis, and cloud-based services. The system establishes separate acquisition channels for radar, RGB video, infrared video, environment, posture, ultrasound, encoder, and handheld controllers according to the data source, and adds timestamps, device identifiers, and data quality status before the data enters subsequent modules.

[0063] The hardware collects laser scanning data, video data, environmental parameters, attitude data, ultrasonic distance, and wheel speed / mileage data within the farm. Specifically, laser scanning data is transmitted to the edge computing end via a 2D LiDAR, and video data is transmitted to the edge computing end via a front-facing camera and an infrared PTZ camera. The 2D LiDAR data access node reads the angle, distance, scanning cycle, and effective range according to the driver configuration, discards scan points that exceed the range or are invalid, and publishes the valid scan data to the synchronous positioning and mapping module. The RGB video access node completes frame reading, size unification, and timestamp marking. The same video stream is sent to the HSV visual line following module and the KCF target following module respectively. The two modules only output candidate control quantities in their corresponding operating states. The infrared video access node sends the video stream to the visual analysis and processing link and the video forwarding node of the edge computing end respectively.

[0064] The real-time control unit reads environmental parameters from the environmental transmitter via the RS485 interface, including temperature, humidity, ammonia, hydrogen sulfide, and carbon dioxide data; it reads attitude data from the ICM-20948 via the SPI interface; it controls the HC-SR04 ultrasonic ranging sensor via GPIO; and it reads the pulses and directions of the left and right wheels via the timer encoder interface to obtain wheel speed and mileage data. The real-time control unit performs range checks, outlier removal, moving average or median filtering according to the set sampling period, and generates a data frame containing the sensor number, sampling time, raw value, processed value, and valid markers. PS2 controller takeover status, physical emergency stop status, and fault status are generated independently as discrete state data and are encapsulated by the real-time control unit along with the aforementioned environmental parameters, attitude data, ultrasonic distance, and wheel speed and mileage data before being transmitted to the edge computing unit.

[0065] S104: Generation of Positioning, Path Planning and Navigation Control Quantities

[0066] The synchronous localization and mapping module maintains a grid map and robot pose based on 2D laser scan data and wheel odometer data, and integrates pose information output from the attitude sensor to improve pose estimation under changing mobile platform posture conditions. Map output is used for path planning and cloud display, while pose output is simultaneously fed into the multimodal data aggregation module. The path planning module generates a global path based on the current position and target point, and combines this with local obstacles to generate linear and angular velocities. Ultrasonic ranging is used only for short-range safety assessments and does not directly write abrupt ranging changes into the global map.

[0067] S105: Visual control quantity generation

[0068] HSV visual line following or KCF target following generates candidate velocity correction values ​​based on the current state. The HSV visual line following module sends the image from the front camera to the HSV color space thresholding module, extracts the guide line region, and calculates the lateral deviation of the guide line center relative to the image center, thereby generating the line following velocity correction value. The KCF target following module generates the following velocity correction value based on the target center position and target scale changes. Modules not corresponding to the current state do not obtain control output rights.

[0069] S106: Motion Command Arbitration and Chassis Closed-Loop Execution

[0070] like Figure 3 As shown, the Jetson Nano edge computing terminal selects the only valid upper-level control source from four candidate control sources—autonomous navigation, visual line following, target tracking, and remote manual control—based on the current operating mode. It then applies amplitude and acceleration limits to the selected linear and angular velocities and performs control timeliness verification. The arbitration result is uniformly represented as chassis linear velocity, angular velocity, and audio-visual control commands, which are sent to the STM32F407 via ROS communication and the USB CDC node.

[0071] System status management is divided into normal operation status management and local security status management according to the processing location. For example... Figure 4 As shown, Jetson Nano is responsible for normal operating states such as standby, autonomous navigation, visual line following, target following, remote manual control, and task completion, and completes state transitions based on task events, navigation status, target status, and remote control timeout; STM32F407 is responsible for PS2 controller takeover, safe stop, and emergency stop, and performs local safety overlay based on controller input, control communication timeout, emergency stop input, and serious chassis failure.

[0072] like Figure 3 As shown, the STM32F407 real-time control terminal performs safety processing on the linear and angular velocities sent by the edge computing terminal, including physical emergency stop processing, PS2 local takeover processing, control communication timeout processing, and chassis fault processing. Then, it calculates the target angular velocities of the left and right wheels according to the kinematic model of the two-wheel differential chassis, and uses incremental PID to perform closed-loop adjustment of wheel speed based on encoder feedback.

[0073] The PS2 wireless receiver's joystick, buttons, and connection status directly enter the STM32F407's local control arbitration and emergency stop safety processing unit. When a valid controller takeover input is detected, the STM32F407 blocks the upper-level control inputs from the Jetson Nano and uses the local control inputs generated by the PS2 controller. After the controller is released, fails, or times out, the STM32F407 first outputs zero speed and enters a safe stop state. After the chassis stops and the recovery conditions are confirmed, the Jetson Nano is then reconfigured into normal operating mode.

[0074] For a two-wheel differential chassis, the STM32F407 calculates the target linear velocity and target angular velocity of the left and right wheel rims using the following formula:

[0075] ;

[0076] in, Indicates the speed along the centerline of the chassis. Indicates the chassis yaw rate. Indicates the center distance between the left and right drive wheels. Indicates the effective radius of the drive wheel. and Indicates the target linear velocity of the left and right wheel rims. and This indicates the target angular velocity of the left and right wheels.

[0077] The left and right wheels use incremental PID to calculate the PWM control input, and the encoder provides feedback on the actual wheel speed.

[0078] ;

[0079] in, This represents the PWM control quantity for the current control cycle. This represents the difference between the target wheel speed and the actual wheel speed. , and These represent the proportional, integral, and differential coefficients, respectively. This indicates the control cycle. Independent error and control outputs are used for the left and right wheels respectively. The controller limits the PWM amplitude and performs fault handling for continuous absence of encoder pulses, sustained speed error exceeding limits, and motor stall.

[0080] S107: Animal behavior category identification, individual segmentation, tracking, and spatial activity analysis

[0081] like Figure 1 As shown, the infrared video stream enters the edge computing terminal, where the edge computing terminal performs instance segmentation, multi-target tracking, and trajectory clustering on the video data.

[0082] Instance segmentation sub-step: Input infrared video frames into the instance segmentation model (the current embodiment uses YOLOv8-seg), and output the target bounding box, instance mask, behavior category, and model confidence frame by frame. The training and testing data used in the current embodiment includes six behavior labels: Lying, Sleeping, Eating, Mouthing, Investigating, and Walking.

[0083] Multi-target tracking sub-step: Predict the current target position based on the target state at the previous moment, maintain the trajectory identifier based on spatial overlap and matching costs, and output the tracking result containing at least the trajectory identifier, timestamp, center point coordinates, and matching state. The current embodiment uses the SORT multi-target tracking algorithm, focusing on the data relationship between detection, identity maintenance, trajectory coordinate transformation, and spatial clustering as key aspects of the system design. The trajectory identifier is used for trajectory association within the current continuous video or the current tracking task and does not represent a permanent animal identity across tasks or after a long interruption.

[0084] The trajectory coordinate transformation sub-step involves converting the pixel coordinates of the tracking center point into the actual planar coordinates of the enclosure based on the camera calibration parameters, robot pose, and enclosure coordinate system. Motion parameters such as distance traveled, velocity, acceleration, and direction change are then calculated accordingly. As the camera moves with the mobile platform, the spatial position of the trajectory points is further updated in conjunction with the robot pose. When the mobile platform's attitude changes significantly, the transformation relationship is updated based on the camera extrinsic parameters and robot pose to reduce the spatial offset introduced by the fixed homography relationship.

[0085] Trajectory clustering sub-step: Density clustering is performed on trajectory points according to a set time window, neighborhood radius, and minimum sample size to identify high-density dwell areas and remove outliers. This embodiment uses the DBSCAN density clustering algorithm. When clustering individual trajectories, the individual dwell areas are output; when merging multiple individual trajectories, the group activity hotspots are output. The two types of results are distinguished by data fields.

[0086] S108: Multimodal Data Association

[0087] The edge computing terminal organizes environmental parameters, robot pose, attitude, wheel speed, mileage, behavior category, trajectory identifier, continuous trajectory, activity hotspots, and device status into a unified record that can be uploaded, cached, and verified.

[0088] Structured inspection records should include at least the following: task identifier, record unique identifier, equipment identifier, enclosure identifier, edge timestamp, robot pose, coordinate system name, environmental parameters, wheel speed and mileage, trajectory points, cluster number, trajectory identifier, behavior category, behavior category confidence level, bounding box, tracking status, data quality marker, algorithm version, and serial number.

[0089] Table 2. Description of Field Grouping and Usage in Structured Inspection Records

[0090]

[0091] As shown in Table 2, data from different sources are grouped and recorded uniformly according to the above fields. When the sampling frequencies of different data sources are inconsistent, alignment is achieved by using the most recent time matching, time window summarization, or interpolation based on the record generation time, and the matching time difference and data quality status are recorded. Data exceeding the allowed time difference is marked as missing or low quality and is not forcibly correlated.

[0092] The robot pose uses a unified map coordinate system, while the animal trajectory uses the enclosure plane coordinate system or a transformed map coordinate system. The location for collecting environmental data is determined based on the robot pose at the corresponding sampling time and associated with the current enclosure identifier, thus supporting queries by individual, region, time period, and task.

[0093] When environmental parameters exceed preset thresholds, or when an individual's activity level, dwell time, and activity area continuously exceed preset deviation thresholds relative to a preset baseline within a preset time window, the edge computing terminal generates an anomaly event record containing the anomaly type, occurrence time, enclosure location, associated trajectory identifier, trigger data, and associated video segment index. The anomaly event record includes the event type, occurrence time, enclosure location, associated trajectory, trigger data, duration, data quality status, and associated video time period or video segment index.

[0094] S109: Real-time upload, cloud storage display, and anomaly alerts

[0095] Structured inspection records are uploaded in real time from the edge computing terminal via MQTT or HTTP, while video data is transmitted independently via RTSP or RTMP, ensuring that the video stream and structured data belong to different links and avoiding the impact of continuous video streams on task instructions and status data. Based on this, the cloud obtains display results such as maps, trajectories, environmental parameters, behavior category records, behavior timelines, activity heatmaps, device status, and abnormal events, and supports the export of structured CSV data.

[0096] S110: Local caching, network recovery and retransmission, and cloud-based deduplication.

[0097] The edge computing terminal completes cloud task distribution, edge result uploading, network status judgment, local caching, recovery and retransmission, and cloud deduplication through structured data communication and network outage retransmission mechanism.

[0098] When the network is normal, structured inspection records are uploaded in real time via MQTT or HTTP, and a confirmation status is returned in the cloud. When the network is interrupted, the edge computing terminal maintains local positioning, motion control, and visual analysis, and writes unconfirmed records to the local cache according to task identifier, record unique identifier, sequence number, and data priority. The cached records are prioritized according to abnormal events, key task records, and regular status data.

[0099] After the network is restored, the edge computing terminal starts retransmitting from the earliest unconfirmed sequence number. The cloud checks whether the record has been entered into the database based on its unique identifier. Records that have not been entered into the database are written to the database and an acknowledgment is returned. Duplicate records are not written again. If the number of consecutive retransmission failures exceeds a set number, the record is retained in the cache and a communication failure status is generated.

[0100] To verify the feasibility and effectiveness of this invention, a prototype was used for testing. According to the test records, the average absolute error of the robot's 2D mapping was approximately 3.2 cm, the average navigation positioning deviation over 10 attempts was approximately 4.7 cm, and it succeeded in all 30 static obstacle avoidance tests and 28 out of 30 dynamic obstacle avoidance tests. The YOLOv8-seg model performed as follows on the current dataset: Mask mAP50 of 0.790, precision of 0.794, and recall of 0.798.

[0101] It should be noted that, in actual implementation, the hardware and software platform and specific algorithms of this invention can be flexibly adjusted according to the scale of the farm, equipment costs, and computational load. For example, the edge computing device is not limited to Jetson Nano and can be replaced by other edge computing platforms with GPUs, NPUs, or artificial intelligence acceleration units; the real-time controller is not limited to STM32F407 and can be replaced by other real-time control chips with timers, serial communication, SPI, and GPIO interfaces; synchronous localization and mapping, HSV visual line following, KCF target following, instance segmentation model, multi-target tracking algorithm, and trajectory clustering method can all be replaced by other localization, navigation, visual analysis, and clustering algorithms with the same input-output relationship. Furthermore, the solution of this invention can be extended to other livestock and poultry farming scenarios such as cattle, sheep, and chickens by adjusting the training data, device size, and navigation parameters. Simultaneously, the edge computing terminal can be equipped with depth cameras, thermal imaging cameras, sound sensors, weighing equipment, water flow sensors, or other environmental sensors, and uniformly associated with them according to task identifiers, device identifiers, trajectory identifiers, timestamps, and robot poses. This terminal can also be further extended to long-term activity trend analysis, establishing a normal activity baseline for individuals or groups by fusing motion trajectories, activity heatmaps, and environmental parameters from multiple consecutive days, and providing alerts for decreased activity, abnormal clustering, and changes in activity areas. The communication link in this invention is not limited to USB CDC; UART, CAN, Ethernet, or other bidirectional serial or network links can be used instead. The uploading of structured inspection data is not limited to MQTT or HTTP; WebSocket or corresponding secure communication methods can also be used, and network outage caching, recovery and retransmission, and cloud deduplication can be completed through serial numbers, unique record identifiers, confirmation status, and retry rules. Furthermore, each layer can connect to aquaculture equipment such as intelligent feeding, automatic watering, pen ventilation, automatic manure removal, and spray disinfection via standardized IoT interfaces. This allows inspection results to generate environmental control or production operation requests, which are then confirmed and executed by management personnel. The layered architecture of this invention can also be extended to the fields of multi-robot task scheduling and aquaculture production collaboration, allocating inspection tasks based on robot location, equipment status, task priority, and the location of abnormal events. It should be noted that the linkage of production equipment and multi-robot collaboration are optional implementation methods and are not essential features of this invention.

[0102] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A smart aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis, characterized in that, include: It provides cloud, edge computing, hardware, and real-time control terminals; The cloud sends inspection tasks to the edge computing terminal. The inspection tasks include task identifier, inspection area, target point, operation mode and security parameters. The edge computing terminal parses the inspection task and configures the corresponding running mode according to the task type; The hardware terminal collects laser scanning data, video data, environmental parameters, posture data, ultrasonic distance, and wheel speed mileage data within the farm, and transmits the laser scanning data and video data to the edge computing terminal, while transmitting the environmental parameters, posture data, ultrasonic distance, and wheel speed mileage data to the real-time control terminal. The edge computing terminal performs synchronous positioning and mapping based on laser scanning data and wheel speed and mileage data to obtain robot pose and map, and generates navigation speed control values ​​based on robot pose, target point and local obstacles in laser scanning data; The real-time control terminal performs safety coverage on the navigation speed control quantity, and then performs differential chassis kinematics calculation and wheel speed closed-loop adjustment; The edge computing terminal performs instance segmentation, multi-target tracking, and trajectory clustering on video data to obtain continuous trajectories and activity hotspots with behavior categories and trajectory identifiers; The edge computing terminal associates environmental parameters, robot pose, device status, behavior category, continuous trajectory, and activity hotspots fed back from the real-time control terminal into structured inspection records; When the network is normal, structured inspection records are uploaded to the cloud in real time; when the network is interrupted, unconfirmed records are written to the local cache. Once the network is restored, the data will be retransmitted according to the serial number, and the cloud will perform deduplication based on the unique identifier of the record.

2. The intelligent aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 1, characterized in that, When the hardware acquires data, it transmits laser scanning data to the edge computing terminal through a two-dimensional LiDAR and transmits video data to the edge computing terminal through a front-facing camera and an infrared PTZ camera. The real-time control terminal reads environmental parameters from the environmental transmitter via the RS485 interface, including temperature, humidity, ammonia, hydrogen sulfide, and carbon dioxide data. It reads attitude data via the SPI interface, outputs a trigger signal via GPIO, and measures the echo pulse width to obtain ultrasonic distance. It also obtains wheel speed and mileage data via the timer encoder interface. The real-time control terminal also generates PS2 handle takeover status, physical emergency stop status, and fault status as discrete status data. It encapsulates the discrete status data along with the environmental parameters, attitude data, ultrasonic distance, and wheel speed and mileage data and transmits them to the edge computing terminal.

3. The intelligent aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 1, characterized in that, The operating modes include autonomous navigation, visual line following, target following, and remote manual control. The edge computing terminal selects the only valid upper-level control source from among the four candidate control sources (autonomous navigation, visual line following, target following, and remote manual control) based on the current operating mode. It then applies amplitude and acceleration limits to the selected linear and angular velocities and performs control timeliness verification. The arbitrated linear and angular velocities are then sent to the real-time control terminal via USB CDC. The real-time control terminal performs safety processing on the linear and angular velocities sent from the edge computing terminal, including physical emergency stop processing, PS2 local takeover processing, control communication timeout processing, and chassis fault processing. It then calculates the target angular velocities of the left and right wheels according to the two-wheel differential chassis kinematic model and uses incremental PID control based on encoder feedback to perform closed-loop wheel speed adjustment.

4. The intelligent aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 1, characterized in that, The edge computing terminal performs instance segmentation, multi-target tracking, and trajectory clustering on video data, specifically including: Infrared video frames are input into the instance segmentation model, and the target bounding box, instance mask, behavior category and model confidence are output frame by frame. Predict the current target position based on the target state of the previous moment, and maintain the trajectory identifier based on spatial overlap and matching cost. Output the tracking result, which includes at least the trajectory identifier, timestamp, center point coordinates and matching state. Based on the camera calibration parameters, robot pose, and enclosure coordinate system, the pixel coordinates of the tracking center point are converted into the actual planar coordinates of the enclosure. The trajectory points are clustered by density according to the set time window, neighborhood radius and minimum number of samples, high-density dwell areas are identified and outliers are removed, and individual dwell areas or group activity hotspots are output.

5. The intelligent aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 1, characterized in that, The structured inspection record includes at least the task identifier, record unique identifier, equipment identifier, enclosure identifier, edge timestamp, robot pose, coordinate system name, environmental parameters, wheel speed and mileage, trajectory points, cluster number, trajectory identifier, behavior category, behavior category confidence level, bounding box, tracking status, data quality marker, algorithm version, and serial number. When generating the structured inspection record, if the sampling frequencies of data from different sources are inconsistent, the edge computing terminal will use the nearest time matching, time window summarization, or interpolation method according to the record generation time to complete the alignment, and record the matching time difference and data quality status.

6. The intelligent aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 1, characterized in that, When environmental parameters exceed preset thresholds, or when an individual's activity level, duration of stay, and area of ​​activity area continuously exceed preset deviation thresholds relative to a preset baseline within a preset time window, the edge computing terminal generates an abnormal event record containing the abnormality type, occurrence time, enclosure location, associated trajectory identifier, trigger data, and associated video clip index.

7. The intelligent aquaculture inspection method based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 1, characterized in that, The structured inspection records are uploaded in real time by the edge computing terminal via MQTT or HTTP, while the video data is transmitted independently via RTSP or RTMP, so that the video stream and the structured data belong to different links; when the network is interrupted, the edge computing terminal writes the unconfirmed records into the local cache according to the task identifier, record unique identifier, sequence number and data priority. After the network is restored, the edge computing terminal re-uploads unconfirmed records according to the sequence number, and the cloud deduplicates the records that have been entered into the database based on the unique identifier of the record.

8. An intelligent aquaculture inspection system based on cloud-edge collaboration and multimodal spatial activity analysis, characterized in that, include: The cloud-based interaction and management module, deployed on a cloud server, is used to issue inspection tasks, receive structured inspection data and write it into the database, and perform deduplication on supplementary records based on the unique identifier of the record. The edge computing and motion control module, deployed on edge computing devices, is used to analyze inspection tasks, complete synchronous localization and mapping to obtain robot pose and map, path planning, visual line following, target following, motion command arbitration, and associate environmental parameters, robot pose, device status, behavior category, continuous trajectory and activity hotspots into structured inspection records based on time and space correlations. The real-time control module, deployed on the real-time controller, has logically independent uplink data channels and downlink control channels with the edge computing device. It is used to collect environmental parameters, attitude data, ultrasonic distance, wheel speed and mileage data and device status, perform safety coverage on the linear velocity and angular velocity sent by the edge computing device, and complete differential chassis kinematics calculation and wheel speed closed-loop adjustment. The hardware sensing and execution module includes a 2D LiDAR, a front-facing camera, an infrared PTZ camera, an environmental transmitter, an attitude sensor, an ultrasonic module, an encoder, a motor driver, and a motor. It is used to acquire laser scanning data, video data, environmental parameters, attitude data, ultrasonic distance, and wheel speed and mileage data, and to execute motion commands.

9. The intelligent aquaculture inspection system based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 8, characterized in that, The edge computing and motion control module includes a motion command arbitration unit, which receives four types of candidate control sources: autonomous navigation, visual line following, target following, and remote manual control. Based on the current operating mode, it selects the only valid upper-level control source, applies amplitude and acceleration limits to the selected linear and angular velocities, performs control timeliness verification, and sends the arbitration-resolved linear and angular velocities to the real-time control module. The real-time control module includes a local control arbitration and emergency stop safety processing unit, which performs safety processing on the linear and angular velocities sent from the edge computing device, including physical emergency stop processing, PS2 local takeover processing, control communication timeout processing, and chassis fault processing. It also calculates the target angular velocities of the left and right wheels according to the two-wheel differential chassis kinematic model and uses incremental PID control based on encoder feedback to perform closed-loop wheel speed adjustment.

10. The intelligent aquaculture inspection system based on cloud-edge collaboration and multimodal spatial activity analysis according to claim 8, characterized in that, The edge computing and motion control module also includes a structured data communication and network outage retransmission unit, which is used to upload structured inspection records in real time when the network is normal, and write unconfirmed records to the local cache according to the task identifier, record unique identifier, sequence number and data priority when the network is interrupted. After the network is restored, the unconfirmed records are retransmitted according to the sequence number, and the cloud performs deduplication on the records already in the database according to the record unique identifier.

Images