Water area monitoring method and system based on semantic driving and time sequence behavior cognition

Abstract

The invention discloses a water area monitoring method and system based on semantic driving and time sequence behavior cognition, and the method achieves the open perception capability of dynamically recognizing unknown foreign matters without retraining through the combination of a multi-modal segmentation large model (SAM3) and a natural language cue word. Motion features are extracted through cross-frame tracking, biological behavior intentions are accurately recognized based on time sequence behavior analysis, real anomalies and environmental interference can be effectively distinguished, and therefore the false alarm rate is greatly reduced; a global dynamic batch processing mechanism is adopted, multiple video frames are combined into a high-dimensional tensor for parallel reasoning, and the GPU utilization rate and the system concurrent processing capacity are remarkably improved; through a video rendering and reasoning asynchronous decoupling architecture, smooth video output can still be kept when the reasoning frame rate is low, and the problem of image jamming caused by introduction of a large model is thoroughly solved; therefore, the method integrates open sensing, high accuracy, high fluency and high concurrent processing, and is very suitable for large-scale application and popularization.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a water monitoring method and system based on semantic-driven and temporal behavior cognition. Background Technology

[0002] Currently, aquaculture and aquatic ecological monitoring mainly rely on manual inspections and regular sampling. In this process, aquaculture personnel need to observe the water surface for a long time to determine the feeding and activity status of organisms and the safety of the aquatic environment. Therefore, this method is not only labor-intensive and subjective, but also difficult to achieve all-weather, standardized, and refined management, often resulting in a delay in the detection of abnormal situations.

[0003] In recent years, deep learning-based computer vision technology has been increasingly applied in this field. Mainstream solutions typically employ object detection networks (such as the YOLO series) or instance segmentation networks (such as Mask R-CNN) to automatically identify monitored targets. However, in real-world high-concurrency and complex scenarios, existing technologies still have the following significant limitations: First, existing models generally suffer from "predefined category limitations" and lack open perception capabilities. Traditional visual models can only recognize specific targets preset during the training phase (such as limited to "fish"). Therefore, when unforeseen abnormal objects appear in the aquaculture environment (such as equipment parts falling off, inspectors' mobile phones accidentally falling into water, invasive alien species, etc.), existing systems often ignore them as background noise due to their lack of semantic understanding. If it is necessary to add the recognition of new targets, it is necessary to go through a time-consuming data collection, annotation and retraining process, which cannot meet the real-time monitoring needs of sudden foreign objects or custom targets in actual scenarios.

[0004] Secondly, existing technologies focus on "static phenotypic analysis" and lack a deep understanding of "temporal behavioral intent." Most existing monitoring technologies (including some segmentation-based methods) mainly focus on surface features in single-frame images (such as identifying white spots and ulcer areas). However, many early abnormalities of organisms (such as surfacing due to hypoxia, panicked swimming due to unsuitable water quality, and solitary swimming) do not have obvious surface lesion features, but are manifested as temporal changes in movement trajectory, speed, and posture. Therefore, it is difficult to capture these dynamic behavioral abnormalities by static image analysis alone, which can easily lead to missed reports or misreporting environmental disturbances (such as swaying aquatic plants) as biological abnormalities.

[0005] Finally, there is a serious "architectural bottleneck" between high-precision models and real-time video stream processing. With the rise of large models in the field of vision, although their recognition accuracy has been significantly improved, the computational overhead is huge. Existing systems mostly adopt a serially coupled architecture of "acquisition-inference-rendering-display". The frame rate of the video stream is directly limited by the inference speed of the model (that is, video frames must wait for the AI ​​model to complete inference before entering the rendering stage, which causes the smoothness (FPS) of the video stream to be forcibly "degraded" to the inference speed of the model). Once a high-precision large model is introduced, the video picture often experiences serious stuttering and latency. In addition, the existing single-channel single-inference mode fails to effectively utilize the parallel computing power of the GPU, making it difficult to achieve high-concurrency real-time monitoring of multiple video streams with limited hardware resources.

[0006] Therefore, given the aforementioned shortcomings, how to provide a water monitoring method based on semantic drive and temporal behavior cognition that has open perception capabilities, high monitoring accuracy, high video rendering smoothness, and high-concurrency real-time monitoring has become an urgent problem to be solved. Summary of the Invention

[0007] The purpose of this invention is to provide a water monitoring method and system based on semantic-driven and temporal behavior cognition, in order to solve the problems of existing technologies that cannot meet the real-time monitoring needs of sudden foreign objects or custom targets in actual scenarios, have poor monitoring accuracy, video rendering stuttering, and cannot achieve high-concurrency real-time monitoring of multiple video streams.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a water monitoring method based on semantic-driven and temporal behavioral cognition is provided, including: Multiple independent acquisition threads are used to acquire video frames captured by various cameras in the water area in parallel and store them in the memory buffer pool corresponding to each camera. According to a preset period, the latest frame at the current moment is read from each memory buffer pool, and high-dimensional tensor batch data is constructed using each latest frame read. The target detection prompt words corresponding to the current monitoring task are determined, and the high-dimensional tensor batch data and target detection prompt words are input into the SAM3 model to obtain the target recognition results of each latest frame; Cross-frame tracking is performed on the targets identified in each latest frame to generate motion feature information corresponding to each target; By utilizing the motion feature information corresponding to each target, the behavioral intent of each target is analyzed to obtain the intent state label of each target. Then, using the intent state label and the target recognition results, the entity state snapshot of the latest frame is generated. For any latest frame, after storing the latest frame for a preset duration, determine whether to generate an entity state snapshot corresponding to the latest frame, wherein the preset duration is less than the acquisition interval of the video frame; If not, obtain the nearest neighbor snapshot, where the nearest neighbor snapshot is the entity state snapshot corresponding to the historical frame with the closest time interval to the latest frame in the memory buffer pool corresponding to any latest frame. Based on the nearest neighbor snapshot and any latest frame, a local monitoring image is generated. After all the latest frames are executed concurrently, the multiple local monitoring images obtained are used to form the water monitoring results at the current moment.

[0009] Based on the aforementioned disclosure, this invention utilizes the generalized semantic capabilities of the Multimodal Segmentation Model (SAM3) to dynamically define target detection categories by injecting natural language prompts. Specifically, when searching for specific items in water, this invention eliminates the need for sample collection, manual annotation, and retraining; it only requires issuing prompts for immediate identification. Therefore, this invention not only accurately segments conventional farmed organisms but also instantly identifies unforeseen, sudden foreign objects without requiring model retraining, achieving a leap from "closed-set recognition" to "open perception." Furthermore, unlike traditional analysis methods that rely solely on single-frame static features, this invention performs cross-frame tracking of the target, extracting its motion feature information across multiple frames, and using this information to identify behavioral intent, thus combining SAM with... 3. The output target recognition results are used to generate entity state snapshots for each frame of the image. This enables the invention to accurately distinguish between "biological stress and panic" and "false movements caused by aquatic plants / light and shadow interference," significantly reducing the false alarm rate in complex environments. Furthermore, the invention uses an active polling mechanism to dynamically assemble the latest frames to be processed from multiple heterogeneous cameras into high-dimensional tensor batches and input them into the GPU for inference, thus significantly improving memory utilization and inference throughput. In addition, the invention constructs independent acquisition threads and concurrent rendering threads, and utilizes a video-first mechanism to generate local monitoring images based on the entity state snapshots of historical frames with the shortest time interval to the video frames when the inference speed of the video frames is slower than the video acquisition speed. Based on this, the smooth output of the monitoring and rendering video can be guaranteed, thereby completely solving the video stuttering problem caused by the introduction of large models. Thus, through the above design, the invention provides a water monitoring technology with open perception capabilities, high monitoring accuracy, high video rendering smoothness, and high concurrency execution, making it very suitable for large-scale application and promotion.

[0010] In one possible design, high-dimensional tensor batch data and object detection cue words are input into the SAM3 model to obtain the object recognition results for each latest frame, including: The high-dimensional tensor batch data and the target detection prompt words are input into the SAM3 model to output the low-resolution feature map corresponding to each latest frame, the segmentation mask box of the target identified in each latest frame in the corresponding low-resolution feature map, and the confidence of the target identified in each latest frame. The segmentation mask boxes in each low-resolution feature map are mapped back to the latest frame corresponding to each low-resolution feature map to obtain the real segmentation mask boxes of the identified targets in each latest frame. The target recognition results for each latest frame are composed of the true segmentation mask bounding boxes and confidence scores of the targets identified in each latest frame.

[0011] In one possible design, targets identified in each of the latest frames are tracked across frames to generate motion feature information for each target, including: For any latest frame, obtain several consecutive historical video frames from the target buffer pool, wherein the target buffer pool is the memory buffer pool of the camera corresponding to any latest frame; Using the target recognition results of several historical video frames and any latest frame, cross-frame trajectory tracking and analysis are performed on the target identified in any latest frame to assign a globally unique ID to the target identified in any latest frame and to construct the corresponding initial motion feature sequence. The initial motion feature sequence is normalized to obtain a motion feature sequence; Using a globally unique ID and a sequence of motion features, motion feature information of the target identified in any of the latest frames is constructed.

[0012] In one possible design, using the target recognition results of several historical video frames and any latest frame, cross-frame trajectory tracking and analysis are performed on the target identified in any latest frame, including: Based on the target recognition result of any latest frame and several historical video frames, and using the ByteTrack algorithm, cross-frame trajectory tracking is performed on the specified target in any latest frame to determine the contour mask of the specified target in each historical video frame, and a globally unique ID is assigned to the specified target in all historical video frames, wherein the specified target is the target identified in any latest frame. Based on the contour mask of the specified target in each historical video frame, multidimensional motion feature discrete solution processing is performed on the specified target to obtain the instantaneous velocity, turning angle, angular velocity and aspect ratio of the specified target in each historical video frame and any latest frame. The instantaneous velocity, turning angle, angular velocity, and aspect ratio of the specified target in each historical video frame and any latest frame are used to form the initial motion feature sequence corresponding to the specified target.

[0013] In one possible design, based on the contour mask of the specified target in each historical video frame, multidimensional motion feature discretization processing is performed on the specified target to obtain the instantaneous velocity, turning angle, angular velocity, and aspect ratio of the specified target in each historical video frame, including: For any given historical video frame, obtain the previous historical video frame and use it as the specified video frame. Based on the contour mask of the specified target in any historical video frame, and the contour mask in the specified video frame, the first centroid coordinates of the specified target in any historical video frame and the second centroid coordinates in the specified video frame are determined. The frame interval duration between any historical video frame and the specified video frame is determined, and the instantaneous velocity of the specified target in any historical video frame is calculated using the first centroid coordinates, the second centroid coordinates, and the frame interval duration. Based on the first centroid coordinates and the second centroid coordinates, the turning angle of the specified target in any historical video frame is calculated; Obtain the turning angle of the specified target in the specified video frame; The angular velocity of the specified target in any historical video frame is calculated using the target's turning angle in that historical video frame and the turning angle in that historical video frame. Determine the minimum bounding rectangle of the contour mask of the specified target in any historical video frame; The aspect ratio of the minimum bounding rectangle is used as the aspect ratio of the specified target in any historical video frame.

[0014] In one possible design, the instantaneous velocity of a specified target in any given historical video frame is calculated using the first centroid coordinates, the second centroid coordinates, and the frame interval duration, including: The instantaneous velocity of the specified target in any historical video frame is calculated using the following formula; ; In the formula, This indicates the instantaneous velocity of the specified target in any of the historical video frames. This represents the x and y coordinates in the first centroid coordinate system. This represents the x and y coordinates in the second centroid coordinate system. Indicates the frame interval duration; The calculation of the turning angle of the specified target in any historical video frame based on the first centroid coordinates and the second centroid coordinates includes: The turning angle of the specified target in any historical video frame is calculated according to the following formula; ; In the formula, This indicates the turning angle of the specified target in any of the historical video frames; Accordingly, calculating the angular velocity of the specified target in any historical video frame using the target's turning angle in that historical video frame and the target's turning angle in the specified video frame includes: The angular velocity of the specified target in any historical video frame is calculated according to the following formula; ; In the formula, This indicates the angular velocity of the specified target in any of the historical video frames. This indicates the turning angle of the specified target in the specified video frame. This indicates the duration of the frame interval.

[0015] In one possible design, each camera corresponds to a separate acquisition thread, and the memory buffer pool for each camera includes: a display buffer and an inference buffer. The video frames captured by each camera within the water area are acquired in parallel and stored in the respective memory buffer pool of each camera, including: The video frames captured by each camera are stored in the display buffer and inference buffer corresponding to each camera, respectively. After storage, the acquisition lock of the acquisition thread corresponding to each camera is released so that the video acquisition frame rate of each camera is kept at the maximum frame rate. This includes reading the latest frame at the current moment from each memory buffer pool according to a preset period, including: According to a preset period, the latest frame at the current moment is read from each buffer to be inferred, so as to construct the high-dimensional tensor batch data using the read latest frames; Accordingly, obtaining the nearest neighbor snapshot includes: Read the historical frame with the closest time interval to the latest frame from the display buffer of the camera corresponding to any latest frame, and use it as the target frame; The entity state snapshot corresponding to the target frame is used as the nearest neighbor snapshot.

[0016] In a possible design, the motion feature information corresponding to any target includes: the globally unique ID corresponding to any target and the motion feature sequence; Specifically, by utilizing the motion feature information corresponding to each target, the behavioral intent of each target is analyzed to obtain the intent state label of each target, including: The motion feature sequence in the motion feature information corresponding to each target is input into the intention cognition and state classification model to obtain the intention state label of each target. The intention cognition and state classification model is a trained LSTM model. Accordingly, using the intent state labels and target recognition results, entity state snapshots for each latest frame are generated, including: Using the intent state label and globally unique ID corresponding to each target, as well as the target recognition results, generate entity state snapshots for each latest frame.

[0017] Secondly, a water monitoring system based on semantic-driven and temporal behavior cognition is provided, including: The distributed image acquisition and flow control unit is used to acquire video frames captured by various cameras in the water area in parallel using multiple independent acquisition threads, and store them in the memory buffer pool corresponding to each camera. The global semantic awareness engine unit is used to read the latest frame at the current moment from each memory buffer pool according to a preset period, and to construct high-dimensional tensor batch data using each latest frame read. The global semantic perception engine unit is also used to determine the target detection prompt words corresponding to the current monitoring task, and input the high-dimensional tensor batch data and target detection prompt words into the SAM3 model to obtain the target recognition results of each latest frame; The temporal behavior cognition unit is used to perform cross-frame tracking of targets identified in each latest frame in order to generate motion feature information corresponding to each target; The temporal behavior cognition unit is also used to analyze the behavioral intentions of each target by utilizing the motion feature information corresponding to each target, obtain the intention state label of each target, and generate the entity state snapshot of each latest frame by using the intention state label and the target recognition results. An asynchronous visualization interaction unit is used to determine whether a snapshot of the entity state corresponding to any latest frame is generated after the latest frame has been stored for a preset duration, wherein the preset duration is less than the acquisition interval of the video frame. If not, the asynchronous visualization interaction unit is used to obtain the nearest neighbor snapshot, where the nearest neighbor snapshot is the entity state snapshot corresponding to the historical frame with the closest time interval to the latest frame in the memory buffer pool corresponding to any latest frame. The asynchronous visualization interaction unit is also used to generate local monitoring images based on the nearest neighbor snapshot and any latest frame, and after all the latest frames have been executed concurrently, to use the obtained multiple local monitoring images to form the water monitoring results at the current moment.

[0018] Thirdly, a water monitoring device based on semantic-driven and temporal behavior cognition is provided. Taking the device as an electronic device as an example, it includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the water monitoring method based on semantic-driven and temporal behavior cognition as described in the first aspect or any possible design of the first aspect.

[0019] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the water monitoring method based on semantic-driven and temporal behavior cognition as described in the first aspect or any possible design of the first aspect.

[0020] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the water monitoring method based on semantic-driven and temporal behavior cognition as described in the first aspect or any possible design of the first aspect.

[0021] Beneficial effects: (1) Extreme concurrency and smoothness brought by asynchronous decoupling architecture: Existing technologies generally adopt a serial coupling architecture of "acquisition-inference-display". The smoothness of video is directly limited by the inference speed of the AI ​​model. This invention benefits from the asynchronous rendering strategy of "video first". No matter how many access channels are increased, the video output stream is always kept at full frame level. Even if the backend AI inference slows down due to increased load, the frontend monitoring screen is still smooth, ensuring high smoothness for users in multi-channel monitoring scenarios.

[0022] (2) Computing power scheduling method based on global dynamic batching: Unlike the traditional mode of allocating independent inference processes to each video stream, this invention constructs a centralized global inference method, that is, through an active polling mechanism, the latest frames to be processed from multiple heterogeneous cameras are dynamically assembled into high-dimensional tensor batches and input into the GPU in parallel for inference; thus, this invention significantly improves the utilization of video memory and inference throughput, and increases the number of high-definition video streams that a single graphics card can support by 3-4 times.

[0023] (3) Zero-cost open foreign object perception capability: Unlike the traditional closed set model that "can only identify fish", this invention utilizes the semantic generalization capability of SAM3 and has the open perception characteristic of "what you think is what you see". That is, when it is necessary to find specific items in the water in special circumstances, this invention does not need to collect samples, manually label, or retrain. It only needs to issue prompt words to achieve instant recognition. In this way, this invention greatly reduces the maintenance cost and response time of the system in dealing with unknown risks.

[0024] (4) Unlike traditional analysis methods that rely solely on static features of a single frame (such as spots and colors), this invention extracts multidimensional motion features of the target within a continuous time window (instantaneous velocity, rate of change of angular velocity, and aspect ratio fluctuations), and uses a long short-term memory network (LSTM) to perform temporal modeling of the feature sequence. In this way, the method can accurately distinguish between "biological stress panic" and "false motion caused by aquatic plants / light and shadow interference", thereby significantly reducing the false alarm rate in complex environments.

[0025] (5) The present invention adopts the "low-resolution calculation and high-resolution mapping" technique, calculates the segmentation mask on the reduced feature map, and then maps it back to the original map. In this way, the post-processing calculation is reduced, thereby further increasing the processing speed.

[0026] (6) Significant hardware cost advantage: Since the present invention significantly improves the utilization rate of video memory and inference throughput, under the same hardware resources (such as a single RTX 4090 / 5090 graphics card), the present invention can smoothly support the real-time analysis of 8-12 channels of high-definition video streams, while traditional technology can only support 1-2 channels. This means that in the same scale of farm deployment, the present invention can save 70%-80% of the computing power hardware procurement cost. Attached Figure Description

[0027] Figure 1 A flowchart illustrating the steps of a water monitoring method based on semantic-driven and temporal behavior cognition provided in an embodiment of the present invention; Figure 2 A global dynamic reasoning flowchart provided for embodiments of the present invention; Figure 3 This is a schematic diagram illustrating the open semantic awareness effect provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the STM timing behavior analysis principle provided in an embodiment of the present invention; Figure 5 A timing diagram for asynchronous "video-first" rendering provided in an embodiment of the present invention; Figure 6 This is a comparison chart of video stream smoothness under multi-channel concurrency provided in an embodiment of the present invention; Figure 7 A comparison chart of false alarm rates under complex aquaculture environments provided in this embodiment of the invention; Figure 8 This is a diagram of an LSTM structure provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of a real-time monitoring screen under normal conditions provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of a real-time monitoring screen under abnormal conditions provided in an embodiment of the present invention; Figure 11This is a structural diagram of a water monitoring system based on semantic-driven and temporal behavior cognition provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0029] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0030] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0031] Example: See Figure 1As shown, the water monitoring method based on semantic-driven and temporal behavior cognition provided in this embodiment can be executed by, but is not limited to, a computer device with certain computing resources and a network camera connected for communication. For example, it can be executed by electronic devices such as servers, edge computers, personal computers (PCs, which are multi-purpose computers of a size, price and performance suitable for personal use; desktop computers, laptops to mini-laptops and tablets and ultrabooks are all personal computers), smartphones or personal digital assistants (PDAs). It is understood that the aforementioned execution entities do not constitute a limitation on the embodiments of this application. Accordingly, the operation steps of this method can be, but are not limited to, the steps S1 to S7 below.

[0032] S1. Multiple independent acquisition threads are used to acquire video frames captured by various cameras in the water area in parallel and store them in the memory buffer pool corresponding to each camera. In this embodiment, for example, but not limited to, an independent stream executor (a tool for executing workflows or task flows, which can accept workflows and data, execute workflows according to predefined steps and conditions, and return execution results) can be configured for each camera. Each stream executor adopts a multi-threaded architecture, including an acquisition thread (which corresponds to a distributed image acquisition and flow control unit) and a visualization thread (which is an asynchronous visualization interaction unit used for subsequent rendering output of monitoring images). In this way, it is equivalent to starting multiple independent acquisition threads (Reader Threads) to continuously read high-definition video streams from various heterogeneous camera devices (such as underwater cameras and PTZ cameras) at the maximum frame rate supported by the camera (such as 30FPS). (That is, one acquisition thread corresponds to one camera) thereby realizing the parallel acquisition of video streams from various cameras in the water area.

[0033] After capturing the corresponding video stream (i.e., several consecutive video frames) using the respective capture threads of each camera, data storage can be performed. In this embodiment, to avoid the backend algorithm blocking video capture, a double buffering mechanism is used to cache the captured video frames. Optionally, for example, the aforementioned memory buffer pool may include, but is not limited to, a display buffer and an inference buffer. Therefore, in this embodiment, the video frames captured by each camera are stored in the display buffer and inference buffer corresponding to each camera, respectively. After storage, the capture lock of the capture thread corresponding to each camera is released to keep the video capture frame rate of each camera at the maximum frame rate (i.e., to ensure that the video capture frame rate (FPS) is always kept at the highest level supported by the device). In this way, setting up two buffers can decouple video capture from algorithm inference.

[0034] In this way, after each acquisition thread stores the video frames it acquires into the corresponding memory buffer pool, a global inference loop can be performed to identify the target to be monitored in the video frames and analyze the behavioral intent. The process can be, but is not limited to, the steps S2 to S5 below.

[0035] S2. According to a preset period, the latest frame at the current moment is read from each memory buffer pool, and the read latest frames are used to construct high-dimensional tensor batch data. In specific implementation, this embodiment adopts an active polling mechanism, periodically extracting the latest frame from the inference buffers corresponding to all stream executors, that is: according to a preset period (e.g., 1 second), the latest frame at the current moment is read from each inference buffer, so as to construct the high-dimensional tensor batch data using the read latest frames; see [link to previous document]. Figure 2 As shown, Figure 2 The dynamic batch assembly in the process packages multiple image frames from different cameras into a high-dimensional tensor batch to maximize the parallel computing efficiency of the GPU. In this way, by assembling the latest frames from different cameras into a high-dimensional tensor batch, the utilization of video memory and inference throughput can be significantly improved, thereby greatly increasing the number of high-definition video streams that a single graphics card can support.

[0036] After the dynamic assembly of multiple latest frame images is completed, target recognition can be performed, as shown in step S3 below.

[0037] S3. Determine the target detection prompt words corresponding to the current monitoring task, and input the high-dimensional tensor batch data and target detection prompt words into the SAM3 model to obtain the target recognition results of each latest frame; In specific applications, this embodiment loads a multimodal segmentation large model (SAM3 model, which is a pre-trained model) as the target detection model, which has two modes: (1) Normal mode: input the prompt word "fish" and output the recognition results corresponding to fish; (2) Open perception mode: when the user sends a custom prompt word, the model uses generalized semantic knowledge to directly locate and segment the unseen foreign objects in the video without retraining; Therefore, different large language prompt words can be set in advance according to different monitoring tasks (i.e., different target detection prompt words are set, such as fish monitoring prompt words and foreign object investigation prompt words), see Figure 2 As shown, in fish monitoring tasks, the general language prompt word can be set to "fish", while in specific foreign object monitoring tasks, the general language prompt word can be set to the name of the foreign object, such as "phone". In this way, different prompt words can be issued according to different application scenarios to achieve real-time identification of specific targets, thereby eliminating the need for sample collection, manual annotation, and retraining, and thus reducing the maintenance cost and response time of the system in dealing with unknown risks.

[0038] Based on this, after obtaining the prompt words, the target can be identified by combining the aforementioned high-dimensional tensor batch data, as shown in steps S31 to S33 below.

[0039] S31. Input the high-dimensional tensor batch data and the target detection prompt words into the SAM3 model to output the low-resolution feature map corresponding to each latest frame, the segmentation mask bounding box of the target identified in each latest frame within the corresponding low-resolution feature map, and the confidence score of the target identified in each latest frame. In specific applications, this embodiment is equivalent to calculating the mask on the low-resolution feature map and then mapping it back to the high-resolution original image to balance accuracy and speed. Therefore, the SAM3 model performs feature extraction on the high-dimensional tensor batch data to obtain the corresponding low-resolution feature map. Then, target detection is performed on the low-resolution feature map to obtain the corresponding segmentation mask and the target confidence score. Finally, the aforementioned low-resolution feature map, segmentation mask bounding box (essentially a detection box), and corresponding confidence score are output. Thus, after obtaining the model output, the segmentation mask bounding box can be mapped, as shown in step S32 below.

[0040] S32. Map the segmentation mask boxes within each low-resolution feature map back to the latest frame corresponding to each low-resolution feature map to obtain the true segmentation mask boxes of the targets identified in each latest frame. In this embodiment, it is possible, but not limited to, upsampling the image to enlarge the low-resolution feature maps to the same size as the original image, and then mapping the segmentation mask boxes back to the original image to obtain the true segmentation mask boxes of the targets (such as fish) identified in each latest frame. Finally, the target confidence score can be combined to generate the corresponding target recognition result, as shown in step S33 below.

[0041] S33. Using the true segmentation mask bounding boxes and confidence scores of the targets identified in each latest frame, construct the target recognition results for each latest frame; in specific implementation; see [link to relevant documentation]. Figure 3 As shown, Figure 3 Figure (a) in the diagram is a schematic diagram of the target recognition result under the aforementioned conventional mode, that is, the recognition result when the input target detection prompt word is "fish"; similarly, Figure 3 Figure (b) shows the recognition result in open sensing mode, and the corresponding target detection prompt word is "phone".

[0042] By utilizing the semantic generalization ability of the SAM3 model through the aforementioned steps S31 to S33, the present invention can have a zero-cost open foreign object perception capability, thus possessing the open perception characteristic of "what you think is what you see". Therefore, the present invention can distinguish between conventional biological targets (Mask A) and untrained foreign object targets (Mask B) without retraining the model for foreign objects.

[0043] After obtaining the target recognition results corresponding to each latest frame, deep behavior analysis can be performed on the target to achieve behavior intent recognition. The process is shown in steps S4 and S5 below.

[0044] S4. Perform cross-frame tracking on the targets identified in each latest frame to generate motion feature information corresponding to each target; in specific applications, this embodiment takes any latest frame as an example to illustrate the process of extracting motion feature information, which may be, but is not limited to, the steps S41 to S44 below.

[0045] S41. For any latest frame, obtain several consecutive historical video frames from the target buffer pool, wherein the target buffer pool is the memory buffer pool of the camera corresponding to any latest frame; in this embodiment, N-1 consecutive frames (e.g., N is 30) preceding any latest frame can be selected from the inference buffer of the target buffer pool as historical video frames; then, cross-frame trajectory association of the target can be performed in these N-1 historical video frames, as shown in step S42 below.

[0046] S42. Using the target recognition results of several historical video frames and any latest frame, perform cross-frame trajectory tracking and analysis on the target identified in any latest frame, so as to assign a globally unique ID to the target identified in any latest frame and construct the corresponding initial motion feature sequence; in this embodiment, the extraction of the initial motion feature sequence mainly includes two processes: cross-frame trajectory association of the target and multi-dimensional motion feature discrete solution processing, and the specific implementation steps may be, but are not limited to, as shown in steps S42a to S42c below.

[0047] S42a. Based on the target recognition result of any latest frame and several historical video frames, and using the ByteTrack algorithm, perform cross-frame trajectory tracking on the specified target in any latest frame to determine the contour mask of the specified target in each historical video frame, and assign a globally unique ID to the specified target in all historical video frames, wherein the specified target is the target identified in any latest frame.

[0048] In practical implementation, the ByteTrack algorithm is used to perform cross-frame association of the target identified in any of the latest frames, thereby achieving identity locking between the specified target and historical trajectories. The specific process of cross-frame trajectory tracking based on the ByteTrack algorithm is as follows: First, Kalman filtering is used to predict the prior position of the trajectory in the current frame. Then, the intersection-union ratio (IOU) cost matrix between the detection box (i.e., the segmentation mask box) output by SAM3 in the current frame and the predicted position is calculated. Finally, the Hungarian algorithm is used to complete the optimal matching, thereby assigning a globally unique ID to the specified target to ensure its identity continuity under occlusion or motion blur. Of course, the ByteTrack algorithm is a commonly used technique for target tracking, and its principle will not be elaborated here.

[0049] Thus, based on the aforementioned ByteTrack algorithm, the tracking of the specified target is completed. After obtaining its contour mask in each historical video frame, the discrete solution of motion features can be performed, as shown in step S42b below.

[0050] S42b. Based on the contour mask of the specified target in each historical video frame, perform multidimensional motion feature discrete calculation on the specified target to obtain the instantaneous velocity, turning angle, angular velocity and aspect ratio of the specified target in each historical video frame and any latest frame. In specific implementation, since the multidimensional motion feature calculation process of the specified target in any latest frame and in each historical video frame is the same, the above calculation process is illustrated below using any historical video frame as an example. It can be, but is not limited to, the steps S42b1 to S42b8 below.

[0051] S42b1. For any historical video frame, obtain the previous historical video frame as the designated video frame; after obtaining the previous historical video frame, the centroid coordinates of the designated target in the aforementioned two video frames can be determined by combining the contour mask of the target, as shown in step S42b2 below.

[0052] S42b2. Based on the contour mask of the specified target in any historical video frame, and the contour mask in the specified video frame, determine the first centroid coordinates and the second centroid coordinates of the specified target in any historical video frame. In this embodiment, the centroid coordinates of the contour mask of the specified target in any historical video frame are used as the first centroid coordinates. Similarly, the centroid coordinates of the contour mask of the specified target in the specified video frame are used as the second centroid coordinates. Then, the instantaneous velocity of the specified target in any historical video frame can be calculated based on the two centroid coordinates, as shown in step S42b3 below.

[0053] S42b3. Determine the frame interval duration between any historical video frame and the specified video frame, and calculate the instantaneous velocity of the specified target in any historical video frame using the first centroid coordinates, the second centroid coordinates, and the frame interval duration; in this embodiment, the instantaneous velocity can be calculated using, but is not limited to, the following formula.

[0054] ; In the formula, This indicates the instantaneous velocity of the specified target in any of the historical video frames. This represents the x and y coordinates in the first centroid coordinate system. This represents the x and y coordinates in the second centroid coordinate system. Indicates the frame interval duration.

[0055] Thus, based on the aforementioned formula, after calculating the instantaneous speed, the steering angle can be calculated, as shown in step S42b4 below.

[0056] S42b4. Calculate the turning angle of the specified target in any historical video frame based on the first centroid coordinates and the second centroid coordinates; in specific applications, for example, but not limited to, the following formula can be used to calculate the aforementioned turning angle.

[0057] ; In the formula, This indicates the turning angle of the specified target in any historical video frame.

[0058] After calculating the turning angle using the aforementioned formula, the angular velocity of the target in any given historical video frame can be calculated by combining the turning angle of the target in the previous historical video frame. The calculation process is shown in steps S42b5 and S42b6 below.

[0059] S42b5. Obtain the turning angle of the specified target in the specified video frame.

[0060] S42b6. Calculate the angular velocity of the specified target in any historical video frame using the target's turning angle in the specified video frame and the turning angle in the specified video frame.

[0061] In this embodiment, the formula for calculating angular velocity may be, but is not limited to, the following: ; In the formula, This indicates the angular velocity of the specified target in any of the historical video frames. This indicates the turning angle of the specified target in the specified video frame. This indicates the duration of the frame interval; in this embodiment, This means taking the modulus of 2π on the calculated angle difference, thereby limiting the angle to between 0 and 2π.

[0062] In this embodiment, angular velocity is used to measure the degree of turning abruptly and is a key indicator for determining the "fright" of the target. When the angular velocity is greater than a threshold (e.g., 15 rad / s), imbalance can be determined. After calculating the angular velocity, the aspect ratio of the target can be calculated, as shown in steps S42b7 and S42b8 below.

[0063] S42b7. Determine the minimum bounding rectangle of the contour mask of the specified target in any historical video frame.

[0064] S42b8. The aspect ratio of the minimum bounding rectangle is used as the aspect ratio of the specified target in any historical video frame; in specific applications, the aspect ratio of the minimum bounding rectangle is used to capture the stretching or curling changes of the specified target during swimming.

[0065] Thus, through the aforementioned steps S42b1 to S42b8, the instantaneous velocity, turning angle, angular velocity, and aspect ratio of the specified target in any of the latest frames and each of the historical video frames can be calculated; then, the aforementioned information can be used to construct the initial motion feature sequence of the specified target, as shown in step S42c below.

[0066] S42c. Using the instantaneous velocity, turning angle, angular velocity and aspect ratio of the specified target in each historical video frame and any latest frame, an initial motion feature sequence corresponding to the specified target is formed.

[0067] After constructing the initial motion feature sequence of the specified target through the aforementioned steps S42a to S42c, the features can be normalized to eliminate individual differences and the influence of dimensions; the normalization process is shown in step S43 below.

[0068] S43. Normalize the initial motion feature sequence to obtain a motion feature sequence. In this embodiment, for example, but not limited to, the Min-Max normalization algorithm can be used to map each feature in the initial motion feature sequence to the interval [0,1]. The normalization object is the instantaneous velocity, angular velocity, and angle in the initial motion feature sequence. After normalization, the aforementioned motion feature sequence can be obtained. For example, each historical video frame corresponds to a feature vector, which is composed of the instantaneous velocity, angular velocity, angle, and aspect ratio of the specified target in the corresponding video frame. A schematic diagram can be found in [reference needed]. Figure 4 As shown; of course, for ease of demonstration, Figure 4 The text only provides three features: velocity, angle, and aspect ratio, omitting angular velocity. Therefore, the entire motion feature sequence is an input row matrix containing N feature vectors, which can be represented as... In the formula, This represents the feature vector corresponding to the specified target in any of the latest frames, while This represents the feature vector corresponding to the specified target in the (t-N+1)th historical video frame.

[0069] After obtaining the motion feature sequence, it can be combined with the globally unique ID of the specified target to form its corresponding motion feature information, as shown in step S44 below.

[0070] S44. Using the globally unique ID and motion feature sequence, construct the motion feature information of the target identified in any latest frame.

[0071] Thus, through the aforementioned steps S41 to S43, a temporal feature sequence of the target can be constructed; then, behavioral intent recognition and state classification can be performed, as shown in step S5 below.

[0072] S5. Utilize the motion feature information corresponding to each target to analyze the behavioral intent of each target, obtain the intent state label of each target, and use the intent state label and the recognition results of each target to generate the entity state snapshot of the latest frame; in specific implementation, for example, but not limited to, inputting the motion feature sequence in the motion feature information corresponding to each target into the intent cognition and state classification model to obtain the intent state label of each target.

[0073] Optionally, for example, the intent recognition and state classification model can be, but is not limited to, a trained LSTM model; see [link to relevant documentation]. Figure 4 As shown, the model analyzes temporal patterns such as sudden speed changes and frequent angle reversals to output the probability distribution of the current target's intent state (i.e., intent state label). The intent state label can be, but is not limited to, "Healthy", "Panic", "Slow or Static", "Balance Loss", or "Noise". When the intent state label is noise, it is filtered in subsequent rendering to solve the problem of high false alarm rate in traditional methods.

[0074] Thus, after obtaining the intent state labels of the targets identified in each latest frame, the entity state snapshots of each latest frame can be generated using the intent state labels and globally unique IDs corresponding to each target, as well as the target identification results. Therefore, an entity state snapshot contains the intent state label, the globally unique ID, and the ground truth segmentation mask box. At the same time, each entity state snapshot is stored in the memory buffer pool corresponding to its respective video frame for subsequent retrieval.

[0075] After obtaining the entity state snapshots corresponding to each latest frame, the monitoring results can be rendered. As mentioned above, each stream executor is configured with a visualization thread. Therefore, the visualization thread corresponding to each camera runs independently and can be executed concurrently. In this embodiment, the principle of "video first" is adopted to render the monitoring images. The process can be, but is not limited to, the steps S6 to S8 below.

[0076] S6. For any latest frame, after storing the latest frame for a preset duration, determine whether a corresponding entity state snapshot has been generated. The preset duration is less than the video frame acquisition interval. In specific implementation, when the acquisition thread of the camera corresponding to the latest frame stores the latest frame in the corresponding display buffer, it immediately starts timing. When the timing reaches the preset duration (e.g., 0.1s, which needs to be less than the video frame acquisition interval to ensure that the rendering start time is earlier than the storage time of the next frame image, thereby preventing the inference result from not corresponding to the image), if there is no corresponding entity state snapshot in the memory buffer corresponding to the latest frame, it means that the AI ​​inference speed is slower than the video acquisition speed. In this case, the entity state snapshot generated at the most recent moment (i.e., the nearest neighbor snapshot) needs to be used to render the current video frame. The process is as shown in step S7 below.

[0077] S7. If not, obtain the nearest neighbor snapshot, wherein the nearest neighbor snapshot is the entity state snapshot corresponding to the historical frame with the closest time interval to the latest frame in the memory buffer pool corresponding to any latest frame; in this embodiment, the historical frame with the closest time interval to the latest frame is read from the display buffer of the camera corresponding to any latest frame and used as the target frame (e.g., if the latest frame is the image at time t, then the historical frame at time t-1 is used as the target frame); then, the entity state snapshot corresponding to the target frame is used as the nearest neighbor snapshot.

[0078] After obtaining the nearest neighbor snapshot, image rendering can be performed, as shown in step S8 below.

[0079] S8. Based on the nearest neighbor snapshot and the latest frame, generate a local monitoring image. After all the latest frames have been executed concurrently, use the obtained multiple local monitoring images to form the water monitoring result at the current moment. In this embodiment, for example, but not limited to, the nearest neighbor snapshot can be superimposed on the latest frame to generate the corresponding local monitoring image. Of course, if the target buffer pool contains an entity state snapshot corresponding to the latest frame after a preset storage time, then the entity state snapshot corresponding to the latest frame is directly superimposed on the latest frame to generate the aforementioned local monitoring image. The rendering timing diagram can be found in [reference needed]. Figure 5 As shown, the visualization thread continuously refreshes the video frame at a fixed high frame rate, while the AI ​​inference thread, due to its large computational load, can update the results at a lower frequency. Therefore, when the visualization thread needs to render the frame at time T, if the latest AI inference result has not yet been generated, it does not wait (does not block), but directly reads the most recently generated "entity state snapshot" and overlays it. This mechanism ensures that even if the AI ​​processing takes a long time, the video output remains continuous and smooth on the client side.

[0080] Furthermore, the generated local monitoring images can be assigned different colors to the actual segmentation mask boxes in the images according to different intent state labels. For example, if the intent state label is healthy, it can be marked as green; if the intent state label is low speed or stagnation, it can be marked as blue; if the intent state label is panic escape, it can be marked as red; and if the intent state label is imbalance, it can be marked as orange.

[0081] Furthermore, this embodiment also includes an abnormal event response and closed-loop feedback. When the LSTM outputs a "Panic" state or the SAM3 detects a specific foreign object such as a "Phone", an alarm logic is immediately triggered, saving a 10-second video clip before and after as evidence, and pushing a warning notification through the Web client. At the same time, the abnormal clip is automatically archived for subsequent incremental fine-tuning of the model.

[0082] In addition, this embodiment also provides a comparison chart of video stream smoothness using the method provided in this embodiment and a traditional serial architecture, which can be seen in the following figure. Figure 6 As shown, Figure 6The diagram illustrates the changes in system video output frame rate (FPS) as the number of connected cameras increases from 1 to 12. The red dashed line (traditional serial architecture) shows a linear, precipitous drop in frame rate. With 4 cameras connected, the FPS drops to 6, resulting in severe stuttering. With 8 cameras connected, the FPS drops to only 3, which is completely insufficient for real-time monitoring. The blue solid line (the method provided in this embodiment), thanks to its asynchronous decoupling architecture, ensures that regardless of the number of cameras connected (within the test range), the video acquisition and rendering threads are not blocked by AI inference speed, maintaining a stable full frame rate of 30 FPS. Therefore, the invention completely solves the video stuttering problem in multi-camera high-concurrency scenarios, achieving high smoothness where "even if the backend computation is slow, the frontend display is not choppy."

[0083] This embodiment also provides a comparison chart of the false alarm rates of this method and traditional methods in complex aquaculture environments. (See attached image) Figure 7 As shown; Figure 7 The paper presents the false alarm rates of traditional methods and the method provided in this embodiment under five typical aquaculture environmental disturbances. The orange bars (traditional static feature analysis) perform extremely poorly in the "aquatic plant swaying" and "light and shadow fluctuation" scenarios, with false alarm rates as high as 18.2% and 14.8%, respectively, easily misinterpreting environmental dynamics as fish anomalies. In contrast, the blue bars (the temporal behavior analysis of this invention): through LSTM modeling of temporal action features, it can effectively distinguish between "regular swimming" and "chaotic background disturbances." Under the same high-interference environment, the false alarm rate of this invention is only 1.2%-1.8%.

[0084] Thus, by introducing behavioral intent cognition in the time dimension, this invention significantly improves the system's anti-interference ability, enabling it to adapt to the complex and ever-changing light and water conditions in real aquaculture environments.

[0085] In practical implementation, to verify the monitoring scheme provided in this embodiment's ability to finely classify the multidimensional behavioral states of organisms (including pathological quiescence, normal roaming, and stress-induced aggression) in complex aquaculture environments, as well as its ability to instantly perceive foreign objects in the open world, a specific application example is also given below: Aquaculture environment: Aquaculture tank measuring 1m × 0.5m × 0.5m.

[0086] Experimental subjects: zebrafish, with an initial average weight of 2g and a total number of 50.

[0087] Vision system: Two 4-megapixel high-definition network cameras, equipped with wide-spectrum LED supplementary lights, are deployed at a depth of 0.4m underwater, one in the middle of the water tank and the other in a corner of the water tank.

[0088] Computing Unit: High-performance edge AI computing box, equipped with an NVIDIA RTX 5090 graphics card (24GB VRAM) and running the Ubuntu 22.04 operating system.

[0089] Software environment: The system comes pre-installed with the Global AI Engine (based on the SAM 3 model compiled with PyTorch) and AIHealthAnalyzer (an LSTM-based behavioral analysis service); the backend server is used for data storage and visualization.

[0090] This system employs the method provided in the first aspect of the aforementioned embodiments to acquire and store video streams captured by various cameras, dynamically assemble multiple batches of video frames, and identify targets. Then, a multi-level feature fusion algorithm is used to perform deep analysis of biological behavior from the physical layer to the semantic layer, constructing motion feature sequences for each fish body, which are then input into an LSTM model for behavioral intent inference. This is achieved through a forgetting gate. Input gate and output gate Iterative calculations to update cell state and hidden state Its structural diagram can be found in [reference]. Figure 8 As shown.

[0091] The update process is as follows: ; ; ; ; ; ; In the formula, This represents the sigmoid activation function. This represents the weight and bias of the forget gate. Input to the model, This is the hidden state from the previous moment. This represents the weights and biases of the input gate. Weights and biases representing cell states, Indicates candidate cell information, This refers to the cell information from the previous time step. This represents the weight and bias of the output gate.

[0092] Thus, by iterating through the three gates in the aforementioned manner, the behavioral probability distribution can be output through a fully connected layer (FC) and a Softmax function, with the following formula: ; In the formula, Represents the probability distribution of behavior. This represents the weights and biases of the fully connected layer.

[0093] Furthermore, in the aforementioned open semantic awareness process where the objective is to be considered, the user-submitted custom prompts... The SAM3 model calculates the cosine similarity matrix between image features and text features of prompt words. : ; In the formula, This represents the similarity between image features and text features. Represents image features.

[0094] In this embodiment, when the similarity exceeds a threshold (e.g., 0.65), a foreign object alarm is activated.

[0095] The monitoring steps are as follows: (1) Initialization: Start the edge computing box, load the sam3 weight file, configure the stream executor to run in Video-First mode, and enable TensorRT acceleration.

[0096] (2) Normal state benchmark test: Under normal conditions, the system records the baseline characteristics of the fish population: average speed. average angular velocity At this time, the fish outline is displayed on the left side of the front-end monitoring interface, and a green tracking box is displayed on the right side. See the diagram below for an illustration. Figure 9 As shown.

[0097] (3) Abnormal behavior detection experiment: Low speed / static detection: When the fish's speed is detected to be too low (below normal), the system uses LSTM to determine the speed, the tracking box on the interface turns blue, and displays "Static (Low Activity)".

[0098] High-speed / Panic Detection: When the fish's speed is detected to be too high (higher than normal), the system uses LSTM to determine the speed, the tracking box on the interface turns red, and the message "Panic (Fast)" is displayed.

[0099] Imbalance detection: When an abnormal angle of the fish body is detected, the system judges based on LSTM, the tracking box on the interface is displayed in orange, and the message "Balance Loss" is displayed.

[0100] If the above abnormal behaviors occur continuously for more than 2 seconds, a warning will appear in the user dialog box, and the corresponding monitoring interface will look like this. Figure 10 As shown.

[0101] Finally, the results and analysis were conducted. The system's early warning status was recorded for 10 consecutive days, and the accuracy was verified by manual sampling. The results analysis table is shown in Table 1 below.

[0102] Table 1 shows the results analysis.

[0103] Table 1

[0104] As can be seen from Table 1 above, the monitoring effect of the present invention on fish is far superior to that of traditional technology.

[0105] Therefore, through the water monitoring method based on semantic-driven and temporal behavior cognition described in detail in steps S1 to S7 above, this invention achieves open perception capability for dynamically identifying unknown foreign objects without retraining by combining a multimodal segmentation large model (SAM3) with natural language prompts, significantly improving system adaptability; by extracting motion features through cross-frame tracking and accurately identifying biological behavioral intentions based on temporal behavior analysis, it can effectively distinguish between real anomalies and environmental interference, thereby greatly reducing the false alarm rate; by adopting a global dynamic batch processing mechanism, multiple video frames are merged into high-dimensional tensors for parallel inference, significantly improving GPU utilization and system concurrent processing capability; through an asynchronous decoupling architecture for video rendering and inference, smooth video output can still be maintained even when the inference frame rate is low, completely solving the problem of screen stuttering caused by the introduction of large models; thus, this invention integrates open perception, high accuracy, high smoothness, and high concurrent processing, making it suitable for large-scale intelligent water monitoring applications and promotion.

[0106] like Figure 11 As shown, the second aspect of this embodiment provides a hardware system for implementing the water monitoring method based on semantic driving and temporal behavior cognition described in the first aspect of the embodiment. The system may include, but is not limited to, a distributed image acquisition and flow control unit, a global semantic perception engine unit, a temporal behavior cognition unit, and an asynchronous visualization interaction unit that are sequentially connected in communication. The distributed image acquisition and flow control unit is used to acquire video frames captured by various cameras in the water area in parallel using multiple independent acquisition threads, and store them in the memory buffer pool corresponding to each camera.

[0107] The global semantic awareness engine unit is used to read the latest frame at the current moment from each memory buffer pool according to a preset period, and to construct high-dimensional tensor batch data using each latest frame read.

[0108] The global semantic perception engine unit is also used to determine the target detection prompt words corresponding to the current monitoring task, and input the high-dimensional tensor batch data and target detection prompt words into the SAM3 model to obtain the target recognition results of each latest frame.

[0109] The temporal behavior cognition unit is used to perform cross-frame tracking of targets identified in each latest frame in order to generate motion feature information corresponding to each target.

[0110] The temporal behavior cognition unit is also used to analyze the behavioral intentions of each target by utilizing the motion feature information corresponding to each target, obtain the intention state label of each target, and generate the entity state snapshot of each latest frame by using the intention state label and the recognition results of each target.

[0111] An asynchronous visualization interaction unit is used to determine whether a snapshot of the entity state corresponding to any latest frame is generated after the latest frame has been stored for a preset duration. The preset duration is less than the acquisition interval of the video frame.

[0112] If not, the asynchronous visualization interaction unit is used to obtain the nearest neighbor snapshot, where the nearest neighbor snapshot is the entity state snapshot corresponding to the historical frame with the closest time interval to the latest frame in the memory buffer pool corresponding to any latest frame.

[0113] The asynchronous visualization interaction unit is also used to generate local monitoring images based on the nearest neighbor snapshot and any latest frame, and after all the latest frames have been executed concurrently, to use the obtained multiple local monitoring images to form the water monitoring results at the current moment.

[0114] The working process, working details and technical effects of the system provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0115] like Figure 12 As shown, the third aspect of this embodiment provides a water area monitoring device based on semantic-driven and temporal behavior cognition. Taking the device as an electronic device as an example, it includes: a memory, a processor, and a transceiver connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the water area monitoring method based on semantic-driven and temporal behavior cognition as described in the first aspect of the embodiment.

[0116] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0117] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0118] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0119] The fourth aspect of this embodiment provides a storage medium that stores instructions containing the water monitoring method based on semantic driving and temporal behavior cognition as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when the instructions are run on a computer, execute the water monitoring method based on semantic driving and temporal behavior cognition as described in the first aspect of the embodiment.

[0120] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0121] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0122] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the water monitoring method based on semantic-driven and temporal behavior cognition as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0123] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A water monitoring method based on semantic-driven and temporal behavioral cognition, characterized in that, include: Multiple independent acquisition threads are used to acquire video frames captured by various cameras in the water area in parallel and store them in the memory buffer pool corresponding to each camera. According to a preset period, the latest frame at the current moment is read from each memory buffer pool, and high-dimensional tensor batch data is constructed using each latest frame read. The target detection prompt words corresponding to the current monitoring task are determined, and the high-dimensional tensor batch data and target detection prompt words are input into the SAM3 model to obtain the target recognition results of each latest frame; Cross-frame tracking is performed on the targets identified in each latest frame to generate motion feature information corresponding to each target; By utilizing the motion feature information corresponding to each target, the behavioral intent of each target is analyzed to obtain the intent state label of each target. Then, using the intent state label and the target recognition results, the entity state snapshot of the latest frame is generated. For any latest frame, after storing the latest frame for a preset duration, determine whether to generate an entity state snapshot corresponding to the latest frame, wherein the preset duration is less than the acquisition interval of the video frame; If not, obtain the nearest neighbor snapshot, where the nearest neighbor snapshot is the entity state snapshot corresponding to the historical frame with the closest time interval to the latest frame in the memory buffer pool corresponding to any latest frame. Based on the nearest neighbor snapshot and any latest frame, a local monitoring image is generated. After all the latest frames are executed concurrently, the multiple local monitoring images obtained are used to form the water monitoring results at the current moment.

2. The method according to claim 1, characterized in that, High-dimensional tensor batch data and object detection cue words are input into the SAM3 model to obtain the object recognition results for each latest frame, including: The high-dimensional tensor batch data and the target detection prompt words are input into the SAM3 model to output the low-resolution feature map corresponding to each latest frame, the segmentation mask box of the target identified in each latest frame in the corresponding low-resolution feature map, and the confidence of the target identified in each latest frame. The segmentation mask boxes in each low-resolution feature map are mapped back to the latest frame corresponding to each low-resolution feature map to obtain the real segmentation mask boxes of the identified targets in each latest frame. The target recognition results for each latest frame are composed of the true segmentation mask bounding boxes and confidence scores of the targets identified in each latest frame.

3. The method according to claim 1, characterized in that, Cross-frame tracking is performed on targets identified in each of the latest frames to generate motion feature information corresponding to each target, including: For any latest frame, obtain several consecutive historical video frames from the target buffer pool, wherein the target buffer pool is the memory buffer pool of the camera corresponding to any latest frame; Using the target recognition results of several historical video frames and any latest frame, cross-frame trajectory tracking and analysis are performed on the target identified in any latest frame to assign a globally unique ID to the target identified in any latest frame and to construct the corresponding initial motion feature sequence. The initial motion feature sequence is normalized to obtain a motion feature sequence; Using a globally unique ID and a sequence of motion features, motion feature information of the target identified in any of the latest frames is constructed.

4. The method according to claim 3, characterized in that, Using the target recognition results of several historical video frames and any latest frame, cross-frame trajectory tracking and analysis are performed on the target identified in any latest frame, including: Based on the target recognition result of any latest frame and several historical video frames, and using the ByteTrack algorithm, cross-frame trajectory tracking is performed on the specified target in any latest frame to determine the contour mask of the specified target in each historical video frame, and a globally unique ID is assigned to the specified target in all historical video frames, wherein the specified target is the target identified in any latest frame. Based on the contour mask of the specified target in each historical video frame, multidimensional motion feature discrete solution processing is performed on the specified target to obtain the instantaneous velocity, turning angle, angular velocity and aspect ratio of the specified target in each historical video frame and any latest frame. The instantaneous velocity, turning angle, angular velocity, and aspect ratio of the specified target in each historical video frame and any latest frame are used to form the initial motion feature sequence corresponding to the specified target.

5. The method according to claim 4, characterized in that, Based on the contour mask of the specified target in each historical video frame, multidimensional motion feature discrete solution processing is performed on the specified target to obtain the instantaneous velocity, turning angle, angular velocity, and aspect ratio of the specified target in each historical video frame, including: For any given historical video frame, obtain the previous historical video frame and use it as the specified video frame. Based on the contour mask of the specified target in any historical video frame, and the contour mask in the specified video frame, the first centroid coordinates of the specified target in any historical video frame and the second centroid coordinates in the specified video frame are determined. The frame interval duration between any historical video frame and the specified video frame is determined, and the instantaneous velocity of the specified target in any historical video frame is calculated using the first centroid coordinate, the second centroid coordinate, and the frame interval duration. Based on the first centroid coordinates and the second centroid coordinates, the turning angle of the specified target in any historical video frame is calculated; Obtain the turning angle of the specified target in the specified video frame; The angular velocity of the specified target in any historical video frame is calculated using the target's turning angle in that historical video frame and the turning angle in that historical video frame. Determine the minimum bounding rectangle of the contour mask of the specified target in any historical video frame; The aspect ratio of the minimum bounding rectangle is used as the aspect ratio of the specified target in any historical video frame.

6. The method according to claim 5, characterized in that, Using the first centroid coordinates, the second centroid coordinates, and the frame interval duration, the instantaneous velocity of the specified target in any historical video frame is calculated, including: The instantaneous velocity of the specified target in any historical video frame is calculated using the following formula; ； In the formula, This indicates the instantaneous velocity of the specified target in any of the historical video frames. This represents the x and y coordinates in the first centroid coordinate system. This represents the x and y coordinates in the second centroid coordinate system. Indicates the frame interval duration; The calculation of the turning angle of the specified target in any historical video frame based on the first centroid coordinates and the second centroid coordinates includes: The turning angle of the specified target in any historical video frame is calculated according to the following formula; ； In the formula, This indicates the turning angle of the specified target in any of the historical video frames; Accordingly, calculating the angular velocity of the specified target in any historical video frame using the target's turning angle in that historical video frame and the target's turning angle in the specified video frame includes: The angular velocity of the specified target in any historical video frame is calculated according to the following formula; ； In the formula, This indicates the angular velocity of the specified target in any of the historical video frames. This indicates the turning angle of the specified target in the specified video frame. This indicates the duration of the frame interval.

7. The method according to claim 1, characterized in that, Each camera corresponds to a separate acquisition thread, and the memory buffer pool for each camera includes: a display buffer and an inference buffer. Video frames captured by each camera within the water area are acquired in parallel and stored in the respective memory buffer pool of each camera, including: The video frames captured by each camera are stored in the display buffer and inference buffer corresponding to each camera, respectively. After storage, the acquisition lock of the acquisition thread corresponding to each camera is released so that the video acquisition frame rate of each camera is kept at the maximum frame rate. This includes reading the latest frame at the current moment from each memory buffer pool according to a preset period, including: According to a preset period, the latest frame at the current moment is read from each buffer to be inferred, so as to construct the high-dimensional tensor batch data using the read latest frames; Accordingly, obtaining the nearest neighbor snapshot includes: Read the historical frame with the closest time interval to the latest frame from the display buffer of the camera corresponding to any latest frame, and use it as the target frame; The entity state snapshot corresponding to the target frame is used as the nearest neighbor snapshot.

8. The method according to claim 1, characterized in that, The motion feature information corresponding to any target includes: the globally unique ID corresponding to that target and the motion feature sequence; Specifically, by utilizing the motion feature information corresponding to each target, the behavioral intent of each target is analyzed to obtain the intent state label of each target, including: The motion feature sequence in the motion feature information corresponding to each target is input into the intention cognition and state classification model to obtain the intention state label of each target. The intention cognition and state classification model is a trained LSTM model. Accordingly, using the intent state labels and target recognition results, entity state snapshots for each latest frame are generated, including: Using the intent state label and globally unique ID corresponding to each target, as well as the target recognition results, generate entity state snapshots for each latest frame.

9. A water monitoring system based on semantic-driven and temporal behavioral cognition, characterized in that, include: The distributed image acquisition and flow control unit is used to acquire video frames captured by various cameras in the water area in parallel using multiple independent acquisition threads, and store them in the memory buffer pool corresponding to each camera. The global semantic awareness engine unit is used to read the latest frame at the current moment from each memory buffer pool according to a preset period, and to construct high-dimensional tensor batch data using each latest frame read. The global semantic perception engine unit is also used to determine the target detection prompt words corresponding to the current monitoring task, and input the high-dimensional tensor batch data and target detection prompt words into the SAM3 model to obtain the target recognition results of each latest frame; The temporal behavior cognition unit is used to perform cross-frame tracking of targets identified in each latest frame in order to generate motion feature information corresponding to each target; The temporal behavior cognition unit is also used to analyze the behavioral intentions of each target by utilizing the motion feature information corresponding to each target, obtain the intention state label of each target, and generate the entity state snapshot of each latest frame by using the intention state label and the target recognition results. An asynchronous visualization interaction unit is used to determine whether a snapshot of the entity state corresponding to any latest frame is generated after the latest frame has been stored for a preset duration, wherein the preset duration is less than the acquisition interval of the video frame. If not, the asynchronous visualization interaction unit is used to obtain the nearest neighbor snapshot, where the nearest neighbor snapshot is the entity state snapshot corresponding to the historical frame with the closest time interval to the latest frame in the memory buffer pool corresponding to any latest frame. The asynchronous visualization interaction unit is also used to generate local monitoring images based on the nearest neighbor snapshot and any latest frame, and after all the latest frames have been executed concurrently, to use the obtained multiple local monitoring images to form the water monitoring results at the current moment.

10. A computer program product containing instructions, characterized in that, When the instructions are executed on the computer, the computer performs the water monitoring method based on semantic-driven and temporal behavior cognition as described in any one of claims 1 to 8.

Images