Fish cross-view recognition method and system based on visual model and multi-position cooperation

Abstract

The application discloses a fish cross-view domain recognition method and system based on a visual model and multi-camera cooperation. The application constructs an independent thread pool to perform concurrent sensing on multiple video streams, and introduces a logical frame sequence number checking mechanism in a shared memory pool, so that a deep learning pipeline is activated only when it is detected that the physical timing is increasing. In this way, the GPU load and I / O blocking risk of the edge device are greatly reduced. At the same time, the DINOv2 model is used for target comparison, and the strong representation ability of the DINOv2 model can capture the subtle texture, pattern and morphological differences of fish bodies, thereby significantly improving the individual recognition accuracy. Finally, the global feature map library and the high-dimensional dense features extracted by the DINOv2 model are matched in similarity, so that each target is assigned a globally unique identifier that is not limited by the physical camera position. Based on this, the identity fracture problem of fish across the camera view domain in the traditional technology is completely solved, and the cross-view domain recognition of fish is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method and system for cross-view recognition of fish based on visual models and multi-camera collaboration. Background Technology

[0002] With the popularization of precision aquaculture technology, computer vision-based fish behavior monitoring has become a key aspect of biomass assessment. Currently, most existing technologies adopt a "detection + single-lens tracking" architecture (such as YOLO combined with the DeepSORT tracking algorithm), which can maintain good target consistency under a single viewpoint. In addition, some systems trigger alarms through preset thresholds and display data through traditional graphical interfaces to achieve fish monitoring.

[0003] Among them, the existing fish monitoring technology has the following shortcomings: (1) Cross-view identity breakage problem; in aquaculture ponds or large net cages, the field of view of a single camera is limited. When a fish individual leaves the field of view of a certain camera and enters the field of view of another camera, due to the lack of global feature association, the system will misjudge it as a new target, resulting in overlapping growth index statistics; (2) There are problems such as light dispersion, particle occlusion and high similarity of fish bodies in the underwater environment. The traditional lightweight feature extraction network (Re-ID Head) has insufficient representation ability, making it difficult to capture subtle textures and morphological differences that can distinguish individuals, thus resulting in low recognition accuracy; (3) High-concurrency video stream processing leads to computing power overload and blocking; when multiple high frame rate network video streams are accessed in parallel, the existing technology usually sends the entire frame into the deep learning framework for inference. This will cause serious overload of GPU computing power of edge devices, and I / O blocking will cause system-level frame drops. Therefore, based on the above shortcomings, how to provide a fish cross-view recognition method with high recognition accuracy, cross-view identity association, and avoidance of computing power overload has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for cross-view recognition of fish based on visual models and multi-camera collaboration, in order to solve the problems of cross-view identity fragmentation, low recognition accuracy, and easy occurrence of computing power overload and blockage in the existing technology.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a cross-view recognition method for fish based on visual models and multi-camera collaboration is provided, including: The video frames of each camera are continuously and in parallel acquired by the pull stream daemon thread and assigned sequence number identifiers to generate a video stream for each camera. Each camera corresponds to a pull stream daemon thread. Each video stream is added to the atomic-level logical frame sequence in the system's shared memory. The logical frame sequence number verification mechanism is used to perform physical timing increment verification on the current frame in each video stream in the atomic-level logical frame sequence. The current frame that passes the physical timing increment verification is used as the available frame. Target recognition is performed on each available frame, and spatial candidate anchor boxes corresponding to each target are extracted from each available frame as visual priors for capturing target instance images; Using a large-scale visual model, feature mapping is performed on each visual prior cropped target instance image to obtain several high-dimensional dense feature vectors. The large-scale visual model includes the DINOv2 model. The similarity between each high-dimensional dense feature vector and each sample feature vector in the global feature map library is calculated, and the maximum similarity corresponding to each high-dimensional dense feature vector is determined. Each sample feature vector corresponds to a globally unique identifier. For any high-dimensional dense feature vector, if the maximum similarity corresponding to any high-dimensional dense feature vector is greater than the similarity threshold, then the globally unique identifier of the sample feature vector corresponding to the maximum similarity is assigned to the target corresponding to any high-dimensional dense feature vector. After all high-dimensional dense feature vectors have been traversed, the globally unique identifier of the corresponding target in each available frame is obtained. Based on the assigned globally unique identifier, cross-view recognition of fish is completed, wherein targets with the same globally unique identifier belong to the same fish.

[0006] Based on the above-disclosed content, this invention constructs an independent thread pool for concurrent sensing of multiple video streams and introduces a logical frame sequence number verification mechanism in the shared memory pool. This mechanism dynamically compares the sequence number of the current frame in each video stream with that of the previous processed frame, activating the deep learning pipeline (i.e., performing subsequent target recognition) only when a physical temporal increment is detected. This mechanism avoids directly feeding all frames into the deep learning model, thus significantly reducing the GPU load and I / O blocking risk on edge devices. Simultaneously, the DINOv2 model replaces the traditional lightweight feature extraction network for target comparison, and its powerful representational capabilities can capture subtle textures and markings on fish bodies. The invention also incorporates morphological differences, enabling robust feature vector extraction even in complex underwater environments such as water scattering and particle occlusion, thus significantly improving individual identification accuracy. Finally, by performing similarity matching between the global feature map library and the high-dimensional dense features extracted by the DINOv2 model, a globally unique identifier is assigned to each target, unrestricted by physical camera position. Based on this, the problem of identity fragmentation when fish cross camera fields of view in traditional technologies is completely solved, realizing cross-field recognition of fish. Therefore, this invention provides a fish cross-field recognition technology with high recognition accuracy, cross-field identity association capability, and avoidance of computational overload, making it highly suitable for large-scale application and promotion.

[0007] In one possible design, the sequence number of each video frame in any video stream is incremented sequentially. A logical frame sequence number verification mechanism is used to perform physical temporal increment verification on the current frame within each video stream of the atomic-level logical frame sequence, including: For any video stream in an atomic-level logical frame sequence, obtain the previous frame of the current frame in that video stream; Determine whether the sequence number identifier corresponding to the current frame in any video stream is an adjacent sequence number identifier and whether the sequence number identifier corresponding to the current frame in any video stream is greater than the sequence number identifier corresponding to the previous frame. If so, the current frame in any video stream is determined to have passed the physical timing increment check, and the current frame in any video stream is considered a usable frame.

[0008] In one possible design, after target identification is performed on each available frame, the method further includes: For any available frame, the target identified from that available frame is taken as a local target object; A timing hit counter is constructed for the local target object, and a state machine is instantiated. The state machine is set with a state identifier of the local target object. The state identifier includes a candidate state, a confirmed state, and a demise state. The initial state identifier of the state machine is the candidate state. Acquire several frames of images of the local target object, wherein the several frames of images include any available frame and multiple consecutive frames preceding the any available frame; Obtain the coordinates of the first detection box and the first center point of the local target object in the t-th frame image of several frames, and the coordinates of the second detection box and the second center point in the (t-1)-th frame image, where the initial value of t is the total number of frames; Based on the first detection box, the coordinates of the first center point, the second detection box, and the coordinates of the second center point, determine whether the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image. If so, increment the count in the timing hit counter by 1 and determine whether the count is greater than the count threshold; If not, then t is decremented by 1, and the coordinates of the first detection box and the first center point of the local target object in the t-th frame image of several frames are re-acquired until the count is greater than the counting threshold. Then, the state identifier of the state machine is adjusted from the candidate state to the confirmed state so that when the state identifier of the state machine is read as the confirmed state, the visual prior cropped target instance image corresponding to the local target object is sent into the visual basic large model.

[0009] In one possible design, based on the first detection box, the coordinates of the first center point, the second detection box, and the coordinates of the second center point, it is determined whether the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image, including: Calculate the intersection-union ratio between the first detection box and the second detection box, and calculate the Euclidean distance between the coordinates of the first center point and the coordinates of the second center point; Determine whether the intersection-union ratio is greater than a preset overlap threshold and whether the Euclidean distance is less than a preset distance; If so, it is determined that the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image, and the count in the timing hit counter is incremented by 1; otherwise, the state flag of the state machine is adjusted from the candidate state to the extinction state, and any available frame is deleted from the system shared memory.

[0010] In one possible design, after obtaining the globally unique identifier of the corresponding target in each available frame, the method further includes: Based on the globally unique identifier of the corresponding target in each available frame, a self-collection daemon process is triggered, wherein the self-collection daemon process stores a spatiotemporal diversity attenuation strategy. Based on the globally unique identifier of the target in each available frame, and using a spatiotemporal diversity attenuation strategy, target instance images are extracted from the visual priors of each available frame to filter out difficult case data. Using the selected difficult example data, a difficult example dataset for class balancing is constructed. The difficult example dataset is then used to optimize the visual basic model, resulting in an optimized visual basic model. This optimized model is then used to perform feature mapping on the new visual prior target instance image after receiving a new visual prior cropped target instance image.

[0011] In one possible design, based on the globally unique identifier of the corresponding target in each available frame, and employing a spatiotemporal diversity attenuation strategy, target instance images are extracted from the visual priors corresponding to each available frame, and difficult example data is selected, including: For any available frame, the globally unique identifier of the corresponding target in that available frame is used as the target identifier; Obtain the first allocation timestamp when the target identifier is assigned to the corresponding target in any available frame, and obtain the second allocation timestamp when the target identifier was last assigned; The number of images corresponding to the target identifier is counted from the system's persistent storage. Calculate the difference between the first allocation timestamp and the second allocation timestamp; Determine whether the difference is greater than or equal to the time cooling step threshold, and whether the number of storage units is less than the maximum sample capacity; If so, the visual prior image corresponding to any available frame is cropped to the target instance image and used as a hard case data.

[0012] In one possible design, after obtaining the globally unique identifier of the corresponding target in each available frame, the method further includes: Obtain the multi-view parallel frame rate for all cameras; A set of globally unique identifiers is formed by using the globally unique identifiers of the corresponding targets in each available frame; Get the spatial candidate anchor boxes extracted from the available frames in each video stream at a preset time, or the total number of globally unique identifiers assigned within a preset historical time period; The population density index is determined based on the total number of all spatial candidate anchor frames or assigned globally unique identifiers. A structured temporal tensor is constructed based on multi-view parallel frame rate, globally unique identifier set and population density index; By using structured temporal tensors, a prompting engineering context is generated and transmitted to a large language model, so that the large language model can generate question-and-answer results based on the prompting engineering context after receiving user question-and-answer data.

[0013] Secondly, a cross-view recognition system for fish based on a visual model and multi-camera collaboration is provided, including: The video stream acquisition unit is used to continuously and in parallel acquire video frames from each camera through the pull stream daemon thread and assign sequence number identifiers to generate a video stream for each camera. Each camera corresponds to a pull stream daemon thread. The verification unit is used to add each video stream to the atomic-level logical frame sequence in the system's shared memory, and use the logical frame sequence number verification mechanism to perform physical timing increment verification on the current frame in each video stream in the atomic-level logical frame sequence, so that the current frame that passes the physical timing increment verification is used as a usable frame. The region proposal generation unit is used to identify targets in each available frame and extract spatial candidate anchor boxes corresponding to each target from each available frame as visual priors to extract target instance maps. The high-dimensional feature mapping unit is used to perform feature mapping on each visual prior cropped target instance image using the visual basic large model to obtain several high-dimensional dense feature vectors, wherein the visual basic large model includes the DINOv2 model. The cross-view recognition unit is used to calculate the similarity between each high-dimensional dense feature vector and each sample feature vector in the global feature map library, and to determine the maximum similarity corresponding to each high-dimensional dense feature vector. Each sample feature vector corresponds to a globally unique identifier. The cross-view recognition unit is also used to, for any high-dimensional dense feature vector, if the maximum similarity corresponding to any high-dimensional dense feature vector is greater than the similarity threshold, assign the globally unique identifier of the sample feature vector corresponding to the maximum similarity to the target corresponding to the high-dimensional dense feature vector, and after traversing all high-dimensional dense feature vectors, obtain the globally unique identifier of the corresponding target in each available frame, so as to complete the cross-view recognition of fish based on the assigned globally unique identifier, wherein targets with the same globally unique identifier belong to the same fish.

[0014] Thirdly, a fish cross-view recognition device based on visual models and multi-camera collaboration is provided. Taking the device as an electronic device as an example, it includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the fish cross-view recognition method based on visual models and multi-camera collaboration as described in the first aspect or any possible design of the first aspect.

[0015] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the fish cross-view recognition method based on visual models and multi-camera collaboration as described in the first aspect or any possible design of the first aspect.

[0016] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the fish cross-view recognition method based on visual models and multi-camera collaboration as described in the first aspect or any possible design of the first aspect.

[0017] Beneficial effects: (1) This invention constructs an independent thread pool to perform concurrent perception of multiple video streams and introduces a logical frame sequence number verification mechanism in the shared memory pool. That is, it dynamically compares the sequence number of the current frame in each video stream with the sequence number of the previous processed frame, and only activates the deep learning pipeline (i.e., performs subsequent target recognition) when it detects that the physical temporal increment is met. In this way, this mechanism avoids sending all frames directly into the deep learning model, thereby greatly reducing the GPU load and I / O blocking risk of edge devices. At the same time, the DINOv2 model is used to replace the traditional lightweight feature extraction network for target comparison. Its powerful representation ability can capture the subtle texture, spots and morphology of fish. The differences enable robust feature vectors to be extracted even in complex underwater environments such as water scattering and particle occlusion, thus significantly improving the accuracy of individual identification. Finally, by performing similarity matching between the global feature map library and the high-dimensional dense features extracted by the DINOv2 model, a globally unique identifier that is not limited by physical camera position is assigned to each target. Based on this, the problem of identity fragmentation of fish when crossing camera fields of view in traditional technologies is completely solved, realizing cross-field recognition of fish. Therefore, this invention provides a fish cross-field recognition technology with high recognition accuracy, cross-field identity association, and avoidance of computational overload, making it very suitable for large-scale application and promotion.

[0018] (2) Before feature mapping, the present invention sets up a local target anti-shake and feature injection pre-filtering algorithm based on temporal stability verification. That is, the present invention maintains a state based on temporal life cycle for each detected local target. Only when the same target is stably locked and shows a consistent motion trajectory in N consecutive frames (such as 3 frames or more) is it judged as a "valid entity" and allowed to enter the re-identification comparison stage. In this way, the mechanism effectively filters out the interference of false positives caused by underwater suspended particles and light and shadow flicker, and protects the global feature base from being polluted by noise data.

[0019] (3) The present invention has an automatic acquisition mechanism for difficult example data based on global ID constraints and spatiotemporal diversity attenuation. That is, after re-identifying and confirming the globally unique identifier (Global ID), the self-acquisition daemon process is triggered. This process has a built-in spatiotemporal diversity attenuation strategy. One is the time step attenuation strategy, which ensures the diversity of sample poses by setting a fixed sampling cooling interval. The other is the sample capacity saturation threshold limit, which means that when the acquisition amount of a certain GID reaches the sample capacity threshold (such as 30 images), it will automatically stop. In this way, through the spatiotemporal diversity attenuation strategy, the target instance image can be extracted from the visual prior corresponding to each available frame, and difficult example data can be screened out. Thus, a high-quality dataset with class balance can be built unattended during the system operation, and the feedback optimization of the visual basic large model can be completed. Based on this, the pain point of high cost of underwater long-tail feature sample acquisition is solved.

[0020] (4) This invention captures the structured temporal tensors of the underlying computer vision pipeline in real time (such as multi-view parallel frame rate, global GID set, population density index), formats them into a highly structured prompt engineering context, and passes them through the API to the large language model (such as LLM compatible with the OpenAI protocol), so that the large language model can use this as a basis for logical reasoning. In this way, zero-threshold multi-round anthropomorphic question and answer interaction for users can be realized. Attached Figure Description

[0021] Figure 1 A flowchart illustrating the cross-view recognition method for fish based on visual models and multi-camera collaboration provided in an embodiment of the present invention; Figure 2 A flowchart of a multi-source heterogeneous flow high-frequency cleaning mechanism based on atomic locks provided in an embodiment of the present invention; Figure 3 A flowchart for matching a large visual model with a feature metric provided in an embodiment of the present invention; Figure 4 A flowchart of difficult example data self-collection and LLM multimodal semantic proxy provided in an embodiment of the present invention; Figure 5 This is a schematic diagram comparing the constancy of target identity under different complex scenarios provided in embodiments of the present invention; Figure 6 This is a schematic diagram comparing the concurrent throughput and latency of dual-stream edge nodes according to an embodiment of the present invention. Figure 7 A schematic diagram of the accumulation efficiency index curve of underwater fish long-tail feature dataset provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the image from camera A provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the image displayed by camera B according to an embodiment of the present invention; Figure 10 This is a structural diagram of a fish cross-view recognition system based on a visual model and multi-camera collaboration provided in an embodiment of the present invention. Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0023] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0024] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0025] Example: See Figure 1 As shown, the fish cross-view recognition method based on visual models and multi-camera collaboration provided in the first aspect of this embodiment can be executed by a computer device with certain computing resources, but is not limited to. For example, the computer device can be, but is not limited to, a server, an edge computer, or a personal computer (PC, which refers to a multi-purpose computer of a size, price, and performance suitable for personal use; desktop computers, laptops, mini-laptops, tablets, and ultrabooks are all personal computers). It is understood that the aforementioned execution subject does not constitute a limitation on the embodiments of this application. Accordingly, the operation steps of this method can be, but are not limited to, the steps S1 to S6 below.

[0026] S1. Video frames from each camera are continuously and parallelly acquired by the streaming daemon thread and assigned sequence numbers to generate a video stream for each camera. Each camera corresponds to a streaming daemon thread.

[0027] In practical implementation, to avoid computational overload and congestion caused by high-concurrency video stream processing, this embodiment provides a high-frequency cleaning mechanism for multi-source heterogeneous streams based on atomic locks. (See [link to relevant documentation]). Figure 2 As shown, in this embodiment, an independent streaming daemon thread is built for each camera (which is deployed underwater). Each streaming daemon thread uses cv2.VideoCapture to pull video frames from the corresponding camera in real time (cv2.VideoCapture is a class in the OpenCV library used to process video files or video streams, which can capture frames from files, cameras, or video streams, i.e., physical frames). At the same time, when any streaming daemon thread captures a video frame from the corresponding camera, it assigns a sequentially increasing sequence number to the captured video frame (e.g., the sequence number of the first video frame is 1, the sequence number of the second video frame is 2, and so on). In this way, after assigning sequence numbers to the captured video frames, multiple RTMP streams (RTMP stream refers to a live stream that uses RTMP (Real-Time Messaging Protocol) for data transmission) can be formed, that is, a video stream corresponding to each camera can be formed.

[0028] Subsequently, to avoid the GPU computing power from being wasted, this embodiment maintains an atomic-level logical frame sequence with a mutex lock in the system's shared memory. In actual use, each video stream is added to this atomic-level logical frame sequence, and a logical frame sequence number verification mechanism is used to perform physical timing increment verification of the current frame in each video stream, so that only the current frame that passes the verification is copied into the inference pipeline, that is, sent into the subsequent deep learning model; wherein, the physical timing increment verification process is as shown in step S2 below.

[0029] S2. Add each video stream to the atomic-level logical frame sequence in the system's shared memory, and use the logical frame sequence number verification mechanism to perform physical timing increment verification on the current frame in each video stream in the atomic-level logical frame sequence, so that the current frame that passes the physical timing increment verification is used as the usable frame.

[0030] In specific implementation, taking any video stream in an atomic-level logical frame sequence as an example, the timing verification process of the current frame within it is illustrated as follows: First, the previous frame of the current frame in any video stream can be obtained; then, it is determined whether the sequence number of the current frame in any video stream is adjacent to the sequence number of the previous frame, and whether the sequence number of the current frame in any video stream is greater than the sequence number of the previous frame; if so, the current frame in any video stream is determined to have passed the physical timing increment verification, and is considered a usable frame (e.g., if the sequence number of the current frame is 3, and the sequence number of the previous frame is 2, then the current frame is determined to have undergone physical timing increment and can be considered a usable frame); otherwise, it is determined that no physical timing update has occurred, and it is not sent to the subsequent inference pipeline; it should also be noted that the current frame in any video stream is the latest frame in that video stream.

[0031] Thus, this embodiment constructs an independent thread pool to concurrently perceive multiple video streams and introduces a logical frame sequence number verification mechanism in the shared memory pool. This mechanism dynamically compares the sequence number of the current frame in each video stream with that of the previous processed frame, and only activates the deep learning pipeline (i.e., performs subsequent target recognition) when it detects that the physical timing is increasing. In other words, feature extraction and matching are only performed on valid and available frames. This avoids directly feeding all frames into the deep learning model, significantly reducing the GPU / CPU load and I / O blocking risk of edge devices, and ensuring the real-time performance and stability of high-concurrency video stream processing.

[0032] Therefore, after completing the physical temporal increment verification of the current frame in each video stream in the sequence, the current frame that passes the verification can be sent to the inference pipeline, that is, sent to the deep learning model, to perform target recognition and feature matching, thereby realizing fish body recognition; wherein, this embodiment provides a cascaded extraction network of spatial prior and deep topological semantics for fish body recognition, that is, a two-stage decoupled architecture is adopted to realize cross-view recognition of fish bodies, the process of which is shown in steps S3 to S6 below.

[0033] S3. Perform target recognition on each available frame, and extract the spatial candidate anchor boxes corresponding to each target from each available frame to serve as visual prior target instance images; in this embodiment, the first stage is region proposal generation, see [link to documentation]. Figure 3As shown, for example, but not limited to, a lightweight YOLO network can be used to perform target recognition on each available frame, thereby outputting spatial candidate bounding boxes for each target within the field of view, which serve as the visual prior for capturing the target instance image for each target. Furthermore, to improve the recognition confidence in complex perturbation scenes, this embodiment also includes a local target stabilization and feature injection pre-filtering algorithm based on temporal stability verification (see [link to documentation]). Figure 3 As shown in the figure, before triggering the high-dimensional feature extraction pipeline, this embodiment maintains a state machine based on the time-series life cycle for each detected local target. Only when the same target is stably locked and shows a consistent motion trajectory in N consecutive frames (such as 3 frames or more) is it determined to be a "valid entity" and allowed to enter the high-dimensional feature extraction and re-identification comparison stage.

[0034] Specifically, the pre-filtering algorithm includes three stages: target state initialization, spatial trajectory consistency verification, and state transition and anti-shake weighting. The specific process may be, but is not limited to, the steps shown below.

[0035] Step 1: For any available frame, the target identified from that available frame is taken as the local target object.

[0036] Step 2: Construct a timing hit counter for the local target object and instantiate a state machine. The state machine is configured with a state identifier for the local target object, which includes a candidate state, a confirmed state, and a demise state. The initial state identifier of the state machine is the candidate state. In this embodiment, the timing hit counter is used to record the number of frames in which the local target object is successfully associated consecutively, and its initial value is 1.

[0037] Thus, after constructing the timing hit counter and state machine, spatial trajectory consistency verification can be performed, as shown in steps three through six below.

[0038] Step 3: Obtain several frames of images of the local target object, wherein the several frames of images include any available frame and a series of consecutive frames of images preceding any available frame; in this embodiment, the local target object is obtained from any available frame as the starting point, and the series of consecutive frames of images preceding any available frame are used to form an image set (i.e., the aforementioned several frames of images) for spatial trajectory consistency verification.

[0039] Then, spatial trajectory consistency verification can be performed, as shown in step four below.

[0040] Step 4: Obtain the first detection box and the coordinates of the first center point of the local target object in the t-th frame of several frames, and the coordinates of the second detection box and the second center point in the (t-1)-th frame. The initial value of t is the total number of frames. In this embodiment, initially, the t-th frame is actually any available frame, and the (t-1)-th frame is the previous frame of any available frame. The first detection box in the t-th frame is the spatial candidate anchor box detected by the YOLO network. At the same time, the coordinates of the first center point are actually the center coordinates of the first detection box. When the YOLO network outputs the first detection box, it will also output its corresponding center coordinates.

[0041] Thus, after obtaining the detection box and center point coordinates of the local target object in two adjacent frames, spatial trajectory consistency verification can be performed based on this, as shown in step 5 below.

[0042] Step 5: Based on the first detection box, the coordinates of the first center point, the second detection box, and the coordinates of the second center point, determine whether the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image. In specific implementation, for example, but not limited to, first calculating the intersection-union ratio between the first and second detection boxes, and calculating the Euclidean distance between the coordinates of the first and second center points; then, determining whether the intersection-union ratio is greater than a preset overlap threshold, and whether the Euclidean distance is less than a preset distance; wherein, if the aforementioned conditions are met, it is determined that the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image, and the following Step 6 (i.e., performing count accumulation in the counter) can be executed.

[0043] The formula for calculating the intersection-union ratio is: ; In the formula, This represents the intersection-union ratio (CUP) between the first and second detection boxes. This indicates the second detection box. The first detection box, This represents the intersection area of ​​the first and second detection boxes. This represents the area of ​​the union of the first and second detection boxes.

[0044] Furthermore, the Euclidean distance between the coordinates of the first center point and the coordinates of the second center point is: ; In the formula, Represents the Euclidean distance, and These represent the x-coordinate and y-coordinate of the first center point, respectively. and These represent the x-coordinate and y-coordinate of the second center point, respectively.

[0045] Thus, when If the overlap is greater than the preset overlap threshold (e.g., 0.3) and the Euclidean distance is less than the preset distance (the preset distance represents the maximum physical kinematic limit distance of the local target object between adjacent frames, i.e., it meets the smooth kinematic constraint), then the local target object is determined to exhibit a "consistent motion trajectory" and is successfully associated with the (t-1)th frame. At this time, the count in the counter is incremented by 1, and the process is as shown in step six below.

[0046] Step 6: If so, increment the count in the timing hit counter by 1 and determine whether the count is greater than the count threshold. In this embodiment, when the intersection-union ratio is less than or equal to the preset overlap threshold and / or the Euclidean distance is greater than or equal to the preset distance, it is determined that the local target object does not satisfy the consistent motion trajectory (indicating that the local target object is a single-frame false positive caused by underwater suspended particles or light and shadow refraction), that is, the association is broken. At this time, the state flag of the state machine is adjusted from the candidate state to the extinction state, and any available frame is deleted from the system shared memory.

[0047] Furthermore, after the timing hit counter increments by 1, it can be determined whether the count meets the counting threshold, for example, whether the counting threshold is 3. If the timing hit counter count is less than or equal to 3, then spatial trajectory consistency verification needs to be performed, as shown in step 7 below.

[0048] Step 7: If not, decrement t by 1 and reacquire the coordinates of the first detection box and the first center point of the local target object in the t-th frame of several frames until the count is greater than the count threshold. Then, adjust the state identifier of the state machine from the candidate state to the confirmed state so that when the state identifier of the state machine is read as the confirmed state, the visual prior cropped target instance image corresponding to the local target object is sent into the visual basic large model.

[0049] In this embodiment, when the count in the counter is greater than 3, it means that the local target object is stably locked in the picture for N consecutive frames (such as 3 frames or more) and shows a consistent motion trajectory. At this time, the state flag of its corresponding state machine can be adjusted from the candidate state to the confirmed state, thereby determining that the spatial candidate anchor box corresponding to the local target object is a valid instance. At the same time, this embodiment sets an admission gateway in the inference pipeline. Only when the state machine of the local target object is found to be in the confirmed state is the clipping of the spatial candidate anchor box corresponding to the local target object allowed to be performed, so as to obtain the corresponding visual prior cropped target instance image, and send it to the backend DINOv2 visual big model for high-dimensional feature extraction and cosine metric matching.

[0050] Through the aforementioned pre-filtering, this embodiment effectively filters out false positive interference caused by underwater suspended particles and light flickering, thereby protecting the global feature database from being contaminated by noisy data.

[0051] After extracting the visual prior target instance map corresponding to each target in each available frame, the second stage can be entered, namely: using the DINOv2 visual basic model based on the ViT (Vision Transformer) architecture to realize the feature mapping of each visual prior target instance map, the process is shown in step S4 below.

[0052] S4. Using a large-scale visual model, feature mapping is performed on each visual prior cropped target instance image to obtain several high-dimensional dense feature vectors. The large-scale visual model includes the DINOv2 model. In this embodiment, the DINOv2 model is pre-trained using massive amounts of unsupervised data. Through a multi-head self-attention mechanism, it can adaptively mine the high-order geometric topology and subtle semantic differences of underwater weakly textured targets without relying on large-scale manual annotation. It can map each visual prior cropped target instance image into high-dimensional dense feature vectors with scale invariance and illumination robustness. (See) Figure 3 As shown, this is equivalent to performing 1024-dimensional feature extraction. Based on this, by utilizing the powerful representation capabilities of DINOv2, it is possible to capture subtle textures and morphological differences in fish bodies. This enables the present invention to achieve accurate individual identification even in complex underwater environments such as water scattering and particle occlusion, thus significantly improving the identification accuracy.

[0053] Thus, after feature mapping is completed, cross-view identity constancy mapping in high-dimensional quantity space can be performed, as shown in step S5 below.

[0054] S5. Calculate the similarity between each high-dimensional dense feature vector and each sample feature vector in the global feature map library, and determine the maximum similarity corresponding to each high-dimensional dense feature vector. Each sample feature vector corresponds to a globally unique identifier. In practical applications, this embodiment pre-constructs a global feature map library (i.e., Figure 3The global feature fingerprint database stores multiple sample feature vectors (i.e., high-dimensional dense sample feature vectors), and each sample feature vector is assigned a globally unique identifier (Global ID, GID). Thus, for any high-dimensional dense feature vector extracted from any field of view, the similarity between it and each sample feature vector in the global feature database is calculated in the metric space using cosine distance. Then, based on the similarity, the globally unique identifier of each target is assigned.

[0055] The formula for calculating the similarity between any high-dimensional dense feature vector and any sample feature vector is as follows: ; In the formula, Represents any high-dimensional dense feature vector With any of the sample feature vectors Similarity between them This represents the weight component of any high-dimensional dense feature vector in the k-th dimension. This represents the component of any sample feature vector in the k-th dimension.

[0056] Thus, based on the aforementioned formula, after calculating the similarity between each high-dimensional dense feature vector and each sample feature vector, an adaptive threshold truncation strategy can be used to overcome the physical blind zone limitation of the camera, thereby assigning a globally unique identifier (Global ID, GID) to spatially discrete and temporally discontinuous targets. The process is shown in step S6 below.

[0057] S6. For any high-dimensional dense feature vector, if the maximum similarity corresponding to any high-dimensional dense feature vector is greater than the similarity threshold, then the globally unique identifier of the sample feature vector corresponding to the maximum similarity is assigned to the target corresponding to the high-dimensional dense feature vector. After all high-dimensional dense feature vectors have been traversed, the globally unique identifier of the corresponding target in each available frame is obtained. Based on the assigned globally unique identifier, cross-view recognition of fish is completed. Targets with the same globally unique identifier belong to the same fish.

[0058] In practical implementation, for any high-dimensional dense feature vector, if the maximum similarity among its similarities with the feature vectors of all samples is greater than a preset adaptive truncation threshold, then... If the value is 0.55, then the globally unique identifier of the sample feature vector corresponding to the maximum similarity will be assigned to the target corresponding to any high-dimensional dense feature vector. If the target corresponding to any high-dimensional dense feature vector is fish A, and the globally unique identifier of the sample feature vector with the maximum similarity is 1, then the globally unique identifier of 1 will be assigned to fish A. Of course, the above examples are just examples and are not intended to limit this embodiment.

[0059] Meanwhile, if the maximum similarity corresponding to any high-dimensional dense feature vector is less than or equal to the similarity threshold, then this high-dimensional dense feature vector is added to the global feature map library as a sample feature vector to achieve dynamic updating of the global feature map library; of course, a brand-new globally unique identifier (i.e. different from each of the global unique identifiers in the library) will also be generated for this high-dimensional dense feature vector and assigned to the target corresponding to this high-dimensional dense feature vector.

[0060] Through the aforementioned design, this embodiment utilizes a lightweight YOLO network to extract region priors, then employs the DINOv2 model with the VisionTransformer architecture to perform high-dimensional feature matrix mapping on local image slices and calculates their cosine similarity with the global fingerprint database. Finally, based on the maximum similarity and through an adaptive threshold truncation strategy, it overcomes the physical blind zone limitation of the camera, thereby assigning a globally unique identifier to each target in each available frame, unrestricted by physical camera position. Thus, this invention effectively solves the problem of identity fragmentation for fish across camera fields of view in the traditional "detection + single-lens tracking" architecture, avoiding statistical errors in growth indicators caused by repeated counting.

[0061] In one possible design, the second aspect of this embodiment, based on the first aspect, provides an automated long-tail data acquisition method based on identity constraints to address the problem of sample scarcity under extreme underwater conditions; specifically, see... Figure 4 As shown, this embodiment sets up an asynchronous storage queue (Async Queue). After completing the allocation of globally unique identifiers for targets in each available frame, this embodiment does not directly perform I / O write operations. Instead, it encapsulates the image slice tensor and high-precision timestamp into a task object through the collector.add_task interface and pushes it into the thread-safe Queue asynchronous buffer pool (i.e., the aforementioned asynchronous storage queue) to ensure that it does not occupy the inference cycle of core computing resources.

[0062] Meanwhile, this embodiment includes a "self-collection daemon process," which is triggered using GID as the anchor point (i.e., triggered after allocating a globally unique identifier). This self-collection daemon process continuously polls the asynchronous Queue buffer pool and acquires a mutex when accessing the GID resource pool to ensure thread safety. Simultaneously, the process incorporates a spatiotemporal diversity decay strategy (including limited timestamp steps and saturation truncation of single instance capacity limits). Without interfering with the real-time performance of the main inference, it automatically constructs a class-balanced hard-mining dataset to drive the aforementioned large-scale visual model to perform closed-loop fine-tuning (i.e., feedback optimization).

[0063] The construction process of the aforementioned difficult example dataset is shown in steps S7 to S9 below.

[0064] S7. Based on the globally unique identifier of the corresponding target in each available frame, trigger the self-collection daemon process, wherein the self-collection daemon process stores a spatiotemporal diversity attenuation strategy; in this embodiment, when the inference thread detects that a globally unique identifier has been assigned to the globally unique identifier of the corresponding target in each available frame, the self-collection daemon process can be triggered, and then the visual prior capture target instance image corresponding to each available frame can be read from the aforementioned asynchronous storage queue; finally, the pre-set spatiotemporal diversity attenuation strategy is used to filter difficult example data, the process of which is shown in step S8 below.

[0065] S8. Based on the globally unique identifier of the corresponding target in each available frame, and using a spatiotemporal diversity attenuation strategy, target instance images are extracted from the visual priors corresponding to each available frame, and difficult example data is filtered out. In practical applications, the spatiotemporal diversity attenuation strategy mainly includes cooling step size and capacity stage strategy (see...). Figure 4 As shown in the figure), this embodiment uses the two strategies mentioned above to screen difficult case data, and the process is as shown in steps S81 to S86 below.

[0066] S81. For any available frame, the globally unique identifier of the corresponding target in the available frame is used as the target identifier. In this embodiment, it is assumed that the globally unique identifier of the corresponding target in the available frame is "2". Then, the target identifier is "2". Then, it is necessary to obtain the allocation time when the target identifier "2" is allocated to the target in the available frame, and the allocation time when the target identifier "2" was last allocated to a historically detected fish, so as to filter difficult case data based on the time difference between the two adjacent allocations of the target identifier.

[0067] The process of obtaining the allocated time is shown in step S82 below.

[0068] S82. Obtain the first allocation timestamp when the target identifier is allocated to the corresponding target in any available frame, and obtain the second allocation timestamp when the target identifier was last allocated.

[0069] After obtaining the time difference between two consecutive allocations of the target identifier, it is also necessary to count the number of archived samples corresponding to the target identifier in the system's persistent storage. The process is shown in step S83 below.

[0070] S83. Calculate the number of images corresponding to the target identifier stored in the system persistent storage; In this embodiment, it is assumed that there are 100 images stored in the system persistent storage, and the target in 20 of the 100 images has a globally unique identifier of "2". Then, the number of images stored for the target identifier is 20; Of course, the above example is only an example, and this embodiment is not limited to the above example.

[0071] After determining the allocation time difference and the number of accumulated archived samples for the target, a logical decision can be made by combining the aforementioned cooling step size and capacity cutoff strategy, as shown in steps S84 to S86 below.

[0072] S84. Calculate the difference between the first allocation timestamp and the second allocation timestamp.

[0073] S85. Determine whether the difference is greater than or equal to the time cooling step threshold and whether the number of storage items is less than the maximum sample capacity. In specific implementation, the time cooling step threshold can be set to 1s, but is not limited to 1s, and the maximum sample capacity can be set to 30 (which is a capacity saturation threshold limit to prevent long-tail data class imbalance). Thus, when the difference is greater than or equal to 1s and the number of storage items is less than 30, the visual prior corresponding to any available frame can be used to extract the target instance image as a hard case data (i.e., long-tail data). The process is shown in step S86 below.

[0074] S86. If so, the visual prior image corresponding to any available frame is captured as a hard case data. In this embodiment, after obtaining a hard case data, I / O can be performed to write it to disk, that is, to store it. Of course, if the aforementioned conditions are not met, silent discarding is performed.

[0075] Thus, based on the aforementioned steps S81 to S86, after extracting target instance images from the visual priors corresponding to each available frame and filtering out difficult example data, the filtered data can be used to form a difficult example dataset; then, the difficult example dataset can be used to perform feedback optimization of the aforementioned visual basic large model, the process of which is shown in step S9 below.

[0076] S9. Using the selected difficult example data, construct a difficult example dataset for class balancing. Use the difficult example dataset to optimize the visual basic model, resulting in an optimized visual basic model. After receiving a new visual prior cropped target instance image, use the optimized visual basic model to perform feature mapping on the new visual prior cropped target instance image.

[0077] Thus, through the aforementioned steps S7 to S9, this invention solves the pain point of high cost in acquiring underwater long-tail feature samples, and can build a high-quality dataset with class balance without human intervention during system operation, thereby enabling continuous optimization of large models.

[0078] In one possible design, the third aspect of this embodiment is a further optimization based on the first and second aspects of the embodiment, see [link to relevant documentation]. Figure 4 As shown, this embodiment provides a semantic alignment proxy mechanism from the visual modality to the language domain of a large model. Specifically, it sets up a multimodal dynamic context gateway and captures structured temporal tensors of the computer vision pipeline in real time (such as multi-view parallel frame rate, global GID set, and population density index). Then, it formats these tensors into a highly structured prompt engineering context and passes them through to the large language model (such as an OpenAI protocol-compatible LLM) via API. This allows the large language model to use this as a foundation for logical reasoning, thereby achieving zero-threshold, multi-turn, anthropomorphic question-and-answer communication for users.

[0079] The semantic alignment proxy process can be, but is not limited to, the steps S10 to S15 below.

[0080] S10. Obtain the multi-view parallel frame rate corresponding to all cameras; In this embodiment, for example, but not limited to, the number of physical frames that each camera successfully breaks through the "atomic lock" (i.e., through physical time-series incremental verification) and completes YOLO and DINOv2 full-link inference in real time within a unit time (per second) can be calculated. That is, the number of available frames sent to the DINOv2 model for each video stream within a unit time is recorded, and thus used as the frame rate of each camera; Then, the frame rates of all cameras are used to form the multi-view parallel frame rate. This value directly reflects the computing power health and system load of the edge computing node.

[0081] After completing the acquisition of multi-view parallel frame rate, a globally unique identifier set can be generated, and the population density index can be determined. The process is shown in steps S11 to S13 below.

[0082] S11. Use the globally unique identifiers of the corresponding targets in each available frame to form a set of globally unique identifiers.

[0083] S12. Obtain the number of spatial candidate anchor frames extracted from the available frames in each video stream at a preset time, or the total number of globally unique identifiers assigned within a preset historical time period; In this embodiment, a video stream represents one camera, so at a certain time, the number of available frames input to the YOLO network in a video stream and detected by the YOLO network can be counted, and this number is used as the number of spatial candidate anchor frames detected in the field of view of a single camera at a certain time.

[0084] Meanwhile, assuming the historical preset duration is 3 seconds, the total number of globally unique identifiers of targets in each available frame within these 3 seconds that have completed similarity matching is counted. Thus, the population density index can be determined based on the total number of all acquired spatial candidate anchor boxes or the total number of globally unique identifiers assigned, as shown in step S13 below.

[0085] S13. Based on the total number of all acquired spatial candidate anchor frames or the total number of assigned globally unique identifiers, determine the population density index. In this embodiment, the total number of all acquired spatial candidate anchor frames or the total number of assigned globally unique identifiers is used as the population density index. After obtaining the population density index, a structured temporal tensor can be constructed by combining the aforementioned set of globally unique identifiers and the multi-view parallel frame rate, as shown in step S14 below.

[0086] S14. Based on the multi-view parallel frame rate, the globally unique identifier set, and the population density index, a structured temporal tensor is constructed. In this embodiment, the aforementioned three data are merged to obtain the structured temporal tensor. Then, based on this, a prompting engineering context can be generated, as shown in step S15 below.

[0087] S15. Using structured temporal tensors, a prompting engineering context is generated and transmitted to a large language model. This allows the large language model to generate question-and-answer results based on the prompting engineering context after receiving user question-and-answer data. In practice, the prompting engineering context records multi-view parallel frame rates, globally unique identifiers for different fish body components, and population density indicators. Therefore, upon receiving different question-and-answer data, an efficient multimodal logical interpretation can be provided based on the aforementioned recorded information. This involves outputting and displaying the concurrent monitoring tensor (frame rate, globally unique identifier) ​​to the user, thereby solving the problem of fish farmers being unable to understand multi-dimensional perception data.

[0088] Through the aforementioned steps S11 to S15, this invention encapsulates the real-time low-level perception data from multiple cameras (such as globally unique identifier sets, population density indicators, and multi-view parallel frame rates) into structured semantics, and uses it as the context prompt input for a large model. In this way, natural language parsing and dynamic reasoning question answering based on the monitoring status can be achieved, thereby solving the problem of the semantic gap between multi-dimensional perception data and users in traditional technologies.

[0089] Therefore, through the detailed explanation above of the fish cross-view recognition method based on visual models and multi-camera collaboration, the present invention has the following beneficial effects: This invention deeply integrates concurrency awareness and logical frame sequence number verification mechanisms for multiple RTMP video streams, significantly reducing GPU load and I / O blocking risks on edge devices; simultaneously, it employs DINOv2 based on the ViT (Vision Transformer) architecture. A large-scale visual model is used to achieve high-precision fish identification. Combined with a global feature map library, it innovatively realizes global fingerprint-level mapping of individual fish under multi-camera conditions, solving the problem of cross-view identity fragmentation. Simultaneously, by collecting difficult example data based on global ID constraints and spatiotemporal diversity attenuation, it addresses the pain point of high cost in acquiring underwater long-tail feature samples. Finally, a large language model is introduced as a cognitive agent to achieve natural language parsing and dynamic reasoning question answering based on monitoring status. Therefore, this invention constructs a complete and closed-loop fish cross-view recognition system, from efficient scheduling of the underlying video stream, robust extraction and global association of mid-level visual features, to data-driven self-collection of difficult examples and interaction of the large language model at the upper level. This technical solution not only significantly reduces the burden of edge computing and overcomes the identity consistency bottleneck in multi-camera scenarios, but also achieves unattended high-quality dataset accumulation and intelligent question-answering interaction, providing a high-precision, low-latency, and easily deployable solution for biomass assessment in precision aquaculture, possessing significant engineering application value and promising prospects for promotion.

[0090] In one possible design, the fourth aspect of this embodiment provides an application example of the method described in the first to third aspects of the embodiments.

[0091] Specifically, this embodiment provides a schematic diagram comparing the constancy of target identity under different complex scenarios, such as... Figure 5 As shown, from Figure 5As can be seen, existing target tracking technologies (such as DeepSORT) can maintain their performance in a single field of view, but when "cross-camera switching" or "high-frequency occlusion" occurs, their spatiotemporal continuity-dependent algorithms cause the number of ID switching (IDSwitch) to spike exponentially, resulting in severe duplication of biomass statistics. In contrast, this invention introduces the DINOv2 visual model to extract high-dimensional features for cosine metric matching, achieving an almost zero identity loss rate even when the target undergoes three or more frequent cross-camera switching operations. This demonstrates that this invention still possesses powerful individual fingerprint-level recognition and locking capabilities in discontinuous spatiotemporal physical blind zones; therefore, this invention completely overcomes the limitations of physical blind zones and achieves cross-domain instance-level identity constancy mapping.

[0092] This embodiment also provides a schematic diagram comparing the concurrent throughput and latency of dual-stream nodes at the edge, such as... Figure 6 As shown, from Figure 6 As can be seen, existing technologies, when processing dual or multiple high-definition video streams, suffer from a lack of effective logical scheduling, resulting in end-to-end latency as high as 185ms and a sharp drop in dual-stream FPS to 12 frames per second, causing screen stuttering. In contrast, this invention pioneers a multi-source heterogeneous stream atomic frequency locking and asynchronous verification mechanism, which increases the dual-stream concurrent FPS by more than 3 times (reaching 38 frames per second) while reducing inference latency to 42ms, perfectly ensuring the high-energy-efficiency real-time operation of the system on ordinary edge computing nodes. Therefore, this invention can greatly reduce computing power redundancy and achieve extremely high concurrent throughput of edge devices.

[0093] Furthermore, this embodiment also provides a schematic diagram of the efficiency exponential curve for the accumulation of underwater fish long-tail feature datasets, see [link / reference]. Figure 7 As shown, from Figure 7 As can be seen, traditional methods relying on manual frame extraction and data cleaning are not only costly but also result in extremely low accumulated data volumes over time, making it impossible to support the iteration of the underlying model. This invention innovatively incorporates an asynchronous data self-collection queue based on a globally unique identifier without interfering with the main monitoring logic. During 72 hours of continuous system operation, nearly 6,000 high-value sample slices with diverse poses were automatically deduplicated, cleaned, and stored on disk. This "self-distillation mechanism" directly transforms the operational process into data wealth, fundamentally solving the industry barrier of acquiring long-tail data in complex aquatic scenarios.

[0094] Furthermore, this embodiment also provides the identity constancy association process of the system in a cross-viewpoint environment, as shown below: 1. Initialization and Environment Setup: When the system starts, the fish_gallery.npy fingerprint library file containing the preset high-dimensional feature embedding of individual fish is preloaded through np.load, and multiple RTMP concurrent streaming threads are started simultaneously to ensure that the field of view (FOV) data of cameras in different geographical locations is injected into the memory buffer in real time.

[0095] 2. Local Detection and Temporal Anti-shake: In the monitoring screen of aquaculture pond No. 1, the YOLO engine performs real-time instance localization and assigns local tracking IDs. To filter out false positives caused by underwater ambient light drift, the system performs a "3-frame stability check," meaning that depth feature extraction logic is only triggered after the target is stably locked for 3 consecutive frames.

[0096] 3. High-dimensional representation and similarity measurement: The system inputs the target candidate region into the DINOv2 model to obtain its mapping vector in the feature space. The cosine similarity between this vector and the base database features is calculated. If the measured value (e.g., 0.85) is higher than the preset classification truncation threshold (0.55), the system determines it as a known target and updates its global unique identifier (Global ID) to 2.

[0097] 4. Cross-viewpoint tracing and confirmation: 20 minutes later, when the target swam away from pool 1 and appeared in the view of pool 2, the similarity dropped to 0.78 due to changes in ambient lighting and shooting angle, but was still significantly higher than the threshold. The system automatically achieved identity inheritance through feature alignment, stably displaying "★ VIP Fish-2" in the visualization interface and the backend data chain, successfully closing the loop for target identity tracing under discontinuous viewpoints.

[0098] like Figure 8 and Figure 9 As shown, Figure 8 The image from camera A was displayed. Figure 9 The images shown are from camera B. These two images visually demonstrate the practical performance of this invention in cross-viewpoint individual re-identification (Re-ID) in a multi-camera concurrent collaborative scenario. The two images are screenshots from real-time monitoring footage of cameras deployed in different physical locations.

[0099] As clearly seen in the image, despite significant differences in the shooting angles of the two cameras, the lighting conditions of the water area, and the swimming posture of the fish in the images, the system of this invention is still able to perform accurate measurement matching using the high-dimensional topological features extracted from the underlying DINOv2 visual model. Within an extremely short inference delay, the system successfully and accurately identified the same physical entity, a fish, in two completely different physical viewpoints and assigned it a completely consistent globally unique identifier (Global ID) in both images.

[0100] The results of this engineering test strongly demonstrate that this invention has completely broken through the industry bottleneck of "physical blind spots leading to identity breakage" in traditional monocular / single-view tracking algorithms, and truly achieved multi-view target data alignment and identity constancy locking in complex underwater environments.

[0101] In addition, this embodiment also provides an example of asynchronous automatic data distillation and collection.

[0102] 1. Asynchronous task dispatch: After completing the Global ID allocation, the main inference thread does not directly perform I / O write operations. Instead, it encapsulates the image slice tensor and high-precision timestamp into a task object through the collector.add_task interface and pushes it into the thread-safe Queue asynchronous buffer pool to ensure that the inference cycle does not occupy core computing resources.

[0103] 2. Multiple constraint strategy cleaning: The background daemon thread DataCollectionThread continuously polls the buffer and acquires a mutex when accessing the GID resource pool to ensure thread safety.

[0104] 3. Adaptive sampling decision: The system executes dual filtering logic: Temporal sparsity constraint: Compare the current timestamp with the last collected record of this GID. If the time difference is less than 1.0 second (i.e., it is in the attitude height redundancy period), then silently discard it. Capacity saturation limit: Check the number of samples stored for this GID. If the preset balance threshold (e.g., 30 samples) has been reached, stop further collection of samples for this individual.

[0105] 4. Persistent Archiving: Target images meeting the criteria are physically archived to disk using the naming convention 0001_c1s1_[UUID].jpg. This method ensures that the collected dataset maintains individual diversity while maximally eliminating redundant samples, providing high-quality, imbalanced raw material for online distillation and fine-tuning of the underlying model.

[0106] This solves the industry barrier of difficulty in obtaining long-tail data in complex water scenarios.

[0107] like Figure 10 As shown, the fifth aspect of this embodiment provides a software system for implementing the fish cross-view recognition method based on visual models and multi-camera collaboration described in the first to third aspects of the embodiments, comprising: The video stream acquisition unit is used to continuously and in parallel acquire video frames from each camera through the pull stream daemon thread and assign sequence number identifiers to generate a video stream for each camera. Each camera corresponds to a pull stream daemon thread.

[0108] The verification unit is used to add each video stream to the atomic-level logical frame sequence in the system's shared memory, and use the logical frame sequence number verification mechanism to perform physical timing increment verification on the current frame in each video stream in the atomic-level logical frame sequence, so that the current frame that passes the physical timing increment verification is used as the usable frame.

[0109] The region proposal generation unit is used to identify targets in each available frame and extract spatial candidate anchor boxes corresponding to each target from each available frame as visual priors to extract target instance maps.

[0110] The high-dimensional feature mapping unit is used to perform feature mapping on each visual prior cropped target instance image using a large visual foundation model to obtain several high-dimensional dense feature vectors, wherein the large visual foundation model includes the DINOv2 model.

[0111] The cross-view recognition unit is used to calculate the similarity between each high-dimensional dense feature vector and each sample feature vector in the global feature map library, and to determine the maximum similarity corresponding to each high-dimensional dense feature vector. Each sample feature vector corresponds to a globally unique identifier.

[0112] The cross-view recognition unit is also used to, for any high-dimensional dense feature vector, if the maximum similarity corresponding to any high-dimensional dense feature vector is greater than the similarity threshold, assign the globally unique identifier of the sample feature vector corresponding to the maximum similarity to the target corresponding to the high-dimensional dense feature vector, and after traversing all high-dimensional dense feature vectors, obtain the globally unique identifier of the corresponding target in each available frame, so as to complete the cross-view recognition of fish based on the assigned globally unique identifier, wherein targets with the same globally unique identifier belong to the same fish.

[0113] The working process, working details and technical effects of the system provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0114] like Figure 11 As shown, the sixth aspect of this embodiment provides a fish cross-view recognition device based on visual model and multi-camera collaboration. Taking the device as an electronic device as an example, it includes: a memory, a processor and a transceiver connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the fish cross-view recognition method based on visual model and multi-camera collaboration as described in the first to third aspects of the embodiments.

[0115] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0116] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee transceiver (a low-power LAN protocol based on the IEEE 802.15.4 standard), a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0117] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0118] The seventh aspect of this embodiment provides a storage medium for storing instructions containing the fish cross-view recognition method based on visual models and multi-camera collaboration as described in the first to third aspects of the embodiments. That is, the storage medium stores instructions that, when the instructions are run on a computer, execute the fish cross-view recognition method based on visual models and multi-camera collaboration as described in the first to third aspects of the embodiments.

[0119] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0120] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0121] The eighth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the fish cross-view recognition method based on visual models and multi-camera collaboration as described in the first to third aspects of the embodiments, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0122] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cross-view recognition of fish based on visual models and multi-camera collaboration, characterized in that, include: The video frames of each camera are continuously and in parallel acquired by the pull stream daemon thread and assigned sequence number identifiers to generate a video stream for each camera. Each camera corresponds to a pull stream daemon thread. Each video stream is added to the atomic-level logical frame sequence in the system's shared memory. The logical frame sequence number verification mechanism is used to perform physical timing increment verification on the current frame in each video stream in the atomic-level logical frame sequence. The current frame that passes the physical timing increment verification is used as the available frame. Target recognition is performed on each available frame, and spatial candidate anchor boxes corresponding to each target are extracted from each available frame as visual priors for capturing target instance images; Using a large-scale visual model, feature mapping is performed on each visual prior cropped target instance image to obtain several high-dimensional dense feature vectors. The large-scale visual model includes the DINOv2 model. The similarity between each high-dimensional dense feature vector and each sample feature vector in the global feature map library is calculated, and the maximum similarity corresponding to each high-dimensional dense feature vector is determined. Each sample feature vector corresponds to a globally unique identifier. For any high-dimensional dense feature vector, if the maximum similarity corresponding to any high-dimensional dense feature vector is greater than the similarity threshold, then the globally unique identifier of the sample feature vector corresponding to the maximum similarity is assigned to the target corresponding to any high-dimensional dense feature vector. After all high-dimensional dense feature vectors have been traversed, the globally unique identifier of the corresponding target in each available frame is obtained. Based on the assigned globally unique identifier, cross-view recognition of fish is completed, wherein targets with the same globally unique identifier belong to the same fish.

2. The method according to claim 1, characterized in that, The sequence number of each video frame in any video stream is incremented sequentially. A logical frame sequence number verification mechanism is used to perform physical temporal increment verification on the current frame within each video stream of the atomic-level logical frame sequence, including: For any video stream in an atomic-level logical frame sequence, obtain the previous frame of the current frame in that video stream; Determine whether the sequence number identifier corresponding to the current frame in any video stream is an adjacent sequence number identifier and whether the sequence number identifier corresponding to the current frame in any video stream is greater than the sequence number identifier corresponding to the previous frame. If so, the current frame in any video stream is determined to have passed the physical timing increment check, and the current frame in any video stream is considered a usable frame.

3. The method according to claim 1, characterized in that, After performing target identification on each available frame, the method further includes: For any available frame, the target identified from that available frame is taken as a local target object; A timing hit counter is constructed for the local target object, and a state machine is instantiated. The state machine is set with a state identifier of the local target object. The state identifier includes a candidate state, a confirmed state, and a demise state. The initial state identifier of the state machine is the candidate state. Acquire several frames of images of the local target object, wherein the several frames of images include any available frame and multiple consecutive frames preceding the any available frame; Obtain the coordinates of the first detection box and the first center point of the local target object in the t-th frame image of several frames, and the coordinates of the second detection box and the second center point in the (t-1)-th frame image, where the initial value of t is the total number of frames; Based on the first detection box, the coordinates of the first center point, the second detection box, and the coordinates of the second center point, determine whether the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image. If so, increment the count in the timing hit counter by 1 and determine whether the count is greater than the count threshold; If not, then t is decremented by 1, and the coordinates of the first detection box and the first center point of the local target object in the t-th frame image of several frames are re-acquired until the count is greater than the counting threshold. Then, the state identifier of the state machine is adjusted from the candidate state to the confirmed state so that when the state identifier of the state machine is read as the confirmed state, the visual prior cropped target instance image corresponding to the local target object is sent into the visual basic large model.

4. The method according to claim 3, characterized in that, Based on the first detection box, the coordinates of the first center point, the second detection box, and the coordinates of the second center point, determine whether the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image, including: Calculate the intersection-union ratio between the first detection box and the second detection box, and calculate the Euclidean distance between the coordinates of the first center point and the coordinates of the second center point; Determine whether the intersection-union ratio is greater than a preset overlap threshold and whether the Euclidean distance is less than a preset distance; If so, it is determined that the local target object has a consistent motion trajectory between the t-th frame image and the (t-1)-th frame image, and the count in the timing hit counter is incremented by 1; otherwise, the state flag of the state machine is adjusted from the candidate state to the extinction state, and any available frame is deleted from the system shared memory.

5. The method according to claim 1, characterized in that, After obtaining the globally unique identifier of the corresponding target in each available frame, the method further includes: Based on the globally unique identifier of the corresponding target in each available frame, a self-collection daemon process is triggered, wherein the self-collection daemon process stores a spatiotemporal diversity attenuation strategy. Based on the globally unique identifier of the target in each available frame, and using a spatiotemporal diversity attenuation strategy, target instance images are extracted from the visual priors of each available frame to filter out difficult case data. Using the selected difficult example data, a difficult example dataset for class balancing is constructed. The difficult example dataset is then used to optimize the visual basic model, resulting in an optimized visual basic model. This optimized model is then used to perform feature mapping on the new visual prior target instance image after receiving a new visual prior cropped target instance image.

6. The method according to claim 5, characterized in that, Based on the globally unique identifier of the target in each available frame, and employing a spatiotemporal diversity attenuation strategy, target instance images are extracted from the visual priors corresponding to each available frame, and difficult example data is selected, including: For any available frame, the globally unique identifier of the corresponding target in that available frame is used as the target identifier; Obtain the first allocation timestamp when the target identifier is assigned to the corresponding target in any available frame, and obtain the second allocation timestamp when the target identifier was last assigned; The number of images corresponding to the target identifier is counted from the system's persistent storage. Calculate the difference between the first allocation timestamp and the second allocation timestamp; Determine whether the difference is greater than or equal to the time cooling step threshold, and whether the number of storage units is less than the maximum sample capacity; If so, the visual prior image corresponding to any available frame is cropped to the target instance image and used as a hard case data.

7. The method according to claim 1, characterized in that, After obtaining the globally unique identifier of the corresponding target in each available frame, the method further includes: Obtain the multi-view parallel frame rate for all cameras; A set of globally unique identifiers is formed by using the globally unique identifiers of the corresponding targets in each available frame; Get the spatial candidate anchor boxes extracted from the available frames in each video stream at a preset time, or the total number of globally unique identifiers assigned within a preset historical time period; The population density index is determined based on the total number of all spatial candidate anchor frames or assigned globally unique identifiers. A structured temporal tensor is constructed based on multi-view parallel frame rate, globally unique identifier set and population density index; By using structured temporal tensors, a prompting engineering context is generated and transmitted to a large language model, so that the large language model can generate question-and-answer results based on the prompting engineering context after receiving user question-and-answer data.

8. A cross-view recognition system for fish based on visual models and multi-camera collaboration, characterized in that, include: The video stream acquisition unit is used to continuously and in parallel acquire video frames from each camera through the pull stream daemon thread and assign sequence number identifiers to generate a video stream for each camera. Each camera corresponds to a pull stream daemon thread. The verification unit is used to add each video stream to the atomic-level logical frame sequence in the system's shared memory, and use the logical frame sequence number verification mechanism to perform physical timing increment verification on the current frame in each video stream in the atomic-level logical frame sequence, so that the current frame that passes the physical timing increment verification is used as a usable frame. The region proposal generation unit is used to identify targets in each available frame and extract spatial candidate anchor boxes corresponding to each target from each available frame as visual priors to extract target instance maps. The high-dimensional feature mapping unit is used to perform feature mapping on each visual prior cropped target instance image using the visual basic large model to obtain several high-dimensional dense feature vectors, wherein the visual basic large model includes the DINOv2 model. The cross-view recognition unit is used to calculate the similarity between each high-dimensional dense feature vector and each sample feature vector in the global feature map library, and to determine the maximum similarity corresponding to each high-dimensional dense feature vector. Each sample feature vector corresponds to a globally unique identifier. The cross-view recognition unit is also used to, for any high-dimensional dense feature vector, if the maximum similarity corresponding to any high-dimensional dense feature vector is greater than the similarity threshold, assign the globally unique identifier of the sample feature vector corresponding to the maximum similarity to the target corresponding to the high-dimensional dense feature vector, and after traversing all high-dimensional dense feature vectors, obtain the globally unique identifier of the corresponding target in each available frame, so as to complete the cross-view recognition of fish based on the assigned globally unique identifier, wherein targets with the same globally unique identifier belong to the same fish.

9. An electronic device, characterized in that, include: The system comprises a memory, a processor, and a transceiver connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the fish cross-view recognition method based on visual models and multi-camera collaboration as described in any one of claims 1 to 7.

10. A computer program product containing instructions, characterized in that, When the instructions are executed on the computer, the computer performs the fish cross-view recognition method based on visual models and multi-camera collaboration as described in any one of claims 1 to 7.

Images