A fish video target segmentation method based on semi-supervision and related device

Abstract

This application discloses a semi-supervised fish video target segmentation method and related apparatus, relating to the field of computer vision technology. The method includes: acquiring a fish video and the pixel mask of the first image frame in the fish video; using the first image frame as a memory frame; sequentially using each subsequent image frame as a query frame; using all memory frames and their pixel masks as input; using a memory encoder to determine foreground memory, background memory, and target memory; using the query frame as input; using a query encoder to determine pixel features; performing similarity matching between the pixel features and the foreground and background memories respectively to obtain foreground readout and background readout; using the background readout to suppress the foreground readout to obtain pixel readout; using the pixel readout and target memory as input; using a target encoder and decoder to determine the pixel mask of the query frame, thus completing the segmentation of the query frame. This application can effectively avoid target confusion and improve segmentation accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a semi-supervised fish video target segmentation method and related apparatus. Background Technology

[0002] With the booming development of smart fisheries, achieving precise monitoring and management of aquaculture through automation technology has become an inevitable trend for improving production efficiency and optimizing resource utilization. In this process, how to clearly and stably track highly similar fish individuals with complex movement trajectories is a key bottleneck in achieving precise feeding, behavioral analysis, and density control.

[0003] Video target segmentation technology, as an efficient video analysis method, can accurately segment fish targets frame by frame, providing support for subsequent counting, tracking and behavior analysis. Video target segmentation technology can be divided into three categories according to the degree of human participation: (1) Automatic: no annotation required, highly versatile but with limited accuracy; (2) Interactive: relies on real-time human intervention, with high accuracy but low efficiency; (3) Semi-supervised: only the first frame needs to be annotated, which can significantly reduce the annotation cost while ensuring accuracy. It is especially suitable for dynamic and long-sequence video tasks, showing significant application advantages.

[0004] In recent years, mainstream semi-supervised video object segmentation techniques have largely relied on memory mechanisms. They store segmentation results from historical frames as memory and use the current frame as the query frame, retrieving information from this memory to complete the segmentation. However, their feature learning and propagation mechanisms typically depend on targets with obvious appearance differences. In fish videos, however, high intra-class similarity often exists between individual fish, making it easy for target confusion to occur when directly applied to fish video processing, thus hindering the achievement of high segmentation accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a semi-supervised fish video target segmentation method and related apparatus, which can effectively avoid target confusion and improve segmentation accuracy.

[0006] To achieve the above objectives, this application provides the following solution.

[0007] Firstly, this application provides a semi-supervised fish video target segmentation method, which includes: Obtain a fish video and the pixel mask of the first image frame in the fish video; the fish video includes multiple image frames, and the pixel mask includes the category of each pixel in the image frame, the category including fish target and background; Use the first image frame as the memory frame and the second image frame as the query frame. Using all the memory frames and the pixel mask of all the memory frames as input, the foreground memory, background memory, and target memory are determined by the memory encoder; Using the query frame as input, a query encoder is used to determine pixel features. The pixel features are then matched with the foreground memory and the background memory for similarity to obtain foreground readout and background readout. The background readout is used to suppress the foreground readout to obtain pixel readout. Using the pixel readout and the target memory as input, a target encoder and decoder are used to determine the pixel mask of the query frame. Determine whether the query frame is the last image frame to obtain a first determination result; if the first determination result is no, determine whether the number of image frames between the query frame and the previous memory frame reaches a preset number to obtain a second determination result; if the second determination result is yes, use the query frame as a memory frame, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using all the memory frames and the pixel masks of all the memory frames as input, and using a memory encoder to determine the foreground memory, background memory, and target memory"; if the second determination result is no, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using the query frame as input, and using a query encoder to determine pixel features".

[0008] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the above-described semi-supervised fish video target segmentation method.

[0009] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described semi-supervised fish video target segmentation method.

[0010] According to the specific embodiments provided in this application, this application has the following technical effects.

[0011] This application provides a semi-supervised fish video target segmentation method and related apparatus. It obtains the fish video and the pixel mask of the first image frame in the video. The first image frame is used as a memory frame, and each subsequent image frame is used as a query frame. Using all memory frames and their pixel masks as input, a memory encoder is used to determine the foreground memory, background memory, and target memory. Using the query frames as input, a query encoder is used to determine pixel features. The pixel features are then matched with the foreground memory and background memory for similarity, resulting in foreground readout and background readout. The background readout is used to suppress the foreground readout, resulting in pixel readout. Using pixel readout and target memory as input, the pixel mask of the query frame is determined by the target encoder and decoder to complete the segmentation of the query frame. Through a dual-path memory reading mechanism for foreground and background (i.e., similarity matching with foreground memory and background memory respectively to obtain foreground readout and background readout), background context information is explicitly introduced, thereby overcoming the interference caused by traditional methods that rely solely on foreground memory. At the same time, a background-guided suppression strategy is adopted (i.e., using background readout to suppress foreground readout to obtain pixel readout), which effectively suppresses erroneous responses caused by background noise, significantly improves feature discrimination power, and thus effectively avoids target confusion and improves segmentation accuracy. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is an application environment diagram for a semi-supervised fish video target segmentation method provided in Embodiment 1 of this application.

[0014] Figure 2 This is a flowchart illustrating a semi-supervised fish video target segmentation method provided in Embodiment 1 of this application.

[0015] Figure 3 This is a schematic diagram illustrating the technical route of a semi-supervised fish video target segmentation method provided in Embodiment 1 of this application.

[0016] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 2 of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] Example 1.

[0019] The semi-supervised fish video target segmentation method provided in this application can be applied to, for example... Figure 1 In the application environment shown, the terminal communicates with the server via a network. The data storage system stores the data the server needs to process. The data storage system can be set up independently, integrated into the server, or placed in the cloud or on another server. The terminal can send a target segmentation request to the server. After receiving the request, the server obtains the fish video and the pixel mask of the first image frame in the video; it uses the first image frame as a memory frame and the second image frame as a query frame; using all memory frames and their pixel masks as input, it uses a memory encoder to determine the foreground memory, background memory, and target memory; using the query frame as input, it uses a query encoder to determine pixel features, and performs similarity matching between these features and the foreground and background memories to obtain foreground readout and background readout. The background readout is used to suppress the foreground readout to obtain pixel readout. Using the pixel readout and target memory as input, it uses the target... The encoder and decoder determine the pixel mask of the query frame; determine whether the query frame is the last image frame to obtain a first determination result; if the first determination result is negative, determine whether the number of image frames between the query frame and the previous memory frame reaches a preset number to obtain a second determination result; if the second determination result is positive, use the query frame as a memory frame, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using all memory frames and the pixel masks of all memory frames as input, and using the memory encoder to determine the foreground memory, background memory, and target memory"; if the second determination result is negative, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using the query frame as input, and using the query encoder to determine pixel features".

[0020] In addition, in some embodiments, the semi-supervised fish video target segmentation method can also be implemented by a server or a terminal alone. For example, the terminal can directly process the target segmentation request to be processed, or the server can obtain the target segmentation request to be processed from the data storage system and process it.

[0021] In one exemplary embodiment, such as Figure 2 As shown, a semi-supervised fish video target segmentation method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 The following steps are used as an example of a server in the example.

[0022] Step S1: Obtain the fish video and the pixel mask of the first image frame in the fish video; the fish video includes multiple image frames, and the pixel mask includes the category of each pixel in the image frame, the category including fish target and background.

[0023] Step S2: Use the first image frame as the memory frame and the second image frame as the query frame.

[0024] Step S3: Using all the memory frames and the pixel mask of all the memory frames as input, determine the foreground memory, background memory, and target memory using a memory encoder.

[0025] Step S4: Using the query frame as input, determine pixel features using a query encoder, perform similarity matching between the pixel features and the foreground memory and the background memory respectively to obtain foreground readout and background readout, suppress the foreground readout using the background readout to obtain pixel readout, and use the pixel readout and the target memory as input to determine the pixel mask of the query frame using a target encoder and decoder.

[0026] Step S5: Determine whether the query frame is the last image frame to obtain a first determination result; if the first determination result is negative, determine whether the number of image frames between the query frame and the previous memory frame reaches a preset number to obtain a second determination result; if the second determination result is positive, use the query frame as a memory frame, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using all the memory frames and the pixel masks of all the memory frames as input, and using a memory encoder to determine the foreground memory, background memory, and target memory"; if the second determination result is negative, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using the query frame as input, and using a query encoder to determine pixel features".

[0027] By implementing steps S1 to S5 as described above, this application can effectively avoid target confusion and improve segmentation accuracy.

[0028] Because fish individuals often exhibit high intra-class similarity, problems such as target confusion, target loss, and error accumulation can arise, especially with dynamic backgrounds, occlusion, and high similarity among multiple targets. Currently, there are no effective semi-supervised video target segmentation techniques to achieve high-precision segmentation of fish videos, making it difficult to effectively address the accuracy issues in fish video target segmentation. Therefore, there is an urgent need for a high-precision semi-supervised fish video target segmentation method that can solve problems such as target confusion, target loss, and error accumulation.

[0029] Therefore, to address the aforementioned problems, this embodiment proposes a semi-supervised fish video target segmentation method, such as... Figure 3 As shown, the designed segmentation model includes a foreground-background memory fusion module, an uncertain region redistribution module, a small area noise filtering module, and a local target re-verification module. By inputting the fish video to be segmented into the segmentation model, the problems of target confusion, target loss, and error accumulation can be effectively solved, thereby improving the segmentation accuracy. By designing the foreground-background memory fusion module, the background context information is explicitly introduced through the dual-path memory reading mechanism of the foreground and background, and the background-guided suppression strategy is adopted to effectively suppress the erroneous response caused by background noise, improve the feature discrimination power, and solve the target confusion problem. In order to further alleviate the target confusion, target loss, and error accumulation, a three-stage post-processing strategy is proposed: (1) Uncertain region redistribution, which corrects the segmentation results of low confidence regions, and can solve the target confusion and error accumulation problems; (2) Small area noise filtering, which eliminates isolated noise points, and can solve the target confusion and error accumulation problems; (3) Local target re-verification, which performs secondary local segmentation based on historical reliable features to recover the temporarily lost fish targets, and can solve the target loss problem. Meanwhile, a contrastive loss is introduced during training to bring a fish target (foreground) closer to its corresponding memory prototype in the embedding space, while pushing it further away from the background and other fish targets (foreground), thereby enhancing class discriminability and helping to solve the target confusion problem.

[0030] The following, combined with Figure 3 This embodiment provides a detailed description of a semi-supervised fish video target segmentation method, which includes the following steps.

[0031] (a) Acquisition of fish videos and pixel-level annotation.

[0032] This embodiment obtains the fish video and the pixel mask of the first image frame in the fish video.

[0033] The target area is captured by a camera. The target area contains multiple swimming fish. Each fish is a fish target, and the resulting fish video is divided into multiple image frames.

[0034] Extract image frames from the fish video and perform pixel-level labeling of the fish targets in the first image frame. This process can be done manually, resulting in a pixel mask for the first image frame in the fish video. The pixel mask includes the category of each pixel in the image frame, which includes fish targets and background. It should be noted that different fish targets belong to different categories. Through the pixel mask, we can not only determine whether a pixel is a fish target, but also which type of fish target it belongs to.

[0035] (ii) Memory generation.

[0036] In this embodiment, the first image frame is used as the memory frame, the second image frame is used as the query frame, and all memory frames and their pixel masks are used as input. The memory encoder is used to determine the foreground memory, background memory, and target memory.

[0037] The foreground-background memory fusion module in this embodiment includes a query encoder, a memory encoder, and a background suppression attention. Both the query encoder and the memory encoder can adopt the Res-Net model. The query encoder is used to process the image frame to obtain the pixel features of the image frame. The memory encoder is used to process the image frame and the pixel mask of the image frame to obtain the target feature map of the image frame. Subsequently, the target memory is generated based on the target feature map of the image frame. The background suppression attention is a background-guided suppression strategy used to suppress the foreground readout by utilizing the background readout to obtain the pixel readout.

[0038] Based on this, all memory frames and their pixel masks are encoded into foreground memory, background memory, and target memory. Specifically, the foreground memory stores the original image of the memory frame (i.e., the memory frame itself), pixel features, pixel readouts, and pixel mask information. The background memory stores the original image of the memory frame, pixel features, pixel readouts, and the local safe background mask information for each fish target. Therefore, the foreground and background memories can be obtained by directly storing the relevant information. The dimension of the target memory is the number of pooled weight masks. In the vector dimension, a memory encoder is used to encode features of all memory frames containing the fish target and the pixel masks of all memory frames, resulting in a target feature map. This target feature map is then pooled to generate pooling weight masks. During the pooling process, a foreground-background separation constraint is introduced, allowing different pooling weight masks to focus on either the target region or non-target regions respectively. Using all pooling weight masks, the target feature map is weighted and aggregated in the spatiotemporal dimension to obtain multiple target memory vectors, thus forming the target memory. It should be noted that the process of obtaining the target memory is a mature existing technology and will not be elaborated upon here.

[0039] At this point, the foreground memory includes the memory frame and its pixel features, pixel readouts, and pixel mask; the background memory includes the memory frame and its pixel features, pixel readouts, and a local safe background mask. The pixel features, pixel readouts, and pixel mask are obtained in the same way as the query frame. The calculation method for the local safe background mask includes the following steps.

[0040] (1) For each pixel in the memory frame, determine whether the pixel is a fish target based on the pixel mask. If yes, set the pixel value of the pixel to the first value. If no, set the pixel value of the pixel to the second value to obtain the global foreground region mask.

[0041] For each memory frame, given a pixel mask ,in, The target number of fish. and These represent the height and width of the pixel mask, respectively, with 0 and 1 indicating the background and foreground (fish target), respectively. Clearly, the pixel mask includes... There are several binary masks, each corresponding to a fish target. For each binary mask, if a pixel value is 0, it represents the background; if a pixel value is 1, it represents the fish target corresponding to the binary mask. This is achieved by adjusting the pixel mask along the target dimension. Summing is performed on the above to generate a global foreground region mask. The global foreground region mask identifies whether each pixel belongs to any fish target. If it does, it is 1 (first value); otherwise, it is 0 (second value).

[0042] (2) Dilate the foreground region in the global foreground region mask to obtain the dilated foreground region. The foreground region is the region composed of pixels with the first pixel value and adjacent positions.

[0043] To avoid the background and foreground regions being too close, max pooling is used to dilate the foreground region, causing the boundary pixels of the foreground region to expand outward by 1 pixel, forming a restricted area (i.e., the dilated foreground region). This is the expanded foreground region.

[0044] (3) The region in the global foreground region mask other than the inflated foreground region is used as the safe background region.

[0045] The supplement to the restricted area is the safe background area. This represents a background area that is at a certain distance from fish targets and does not include any fish targets. Pixels within the safe background area have a value of 1, while pixels outside the safe background area have a value of 0.

[0046] (4) For each fish target region in the global foreground region mask, generate the minimum bounding rectangle of the fish target region, enlarge the minimum bounding rectangle to obtain a rectangular safe box, perform Hadamard product operation on the safe background region and the rectangular safe box to obtain the local safe background mask of the fish target region. The fish target region is a region composed of pixels of the same type of fish target and adjacent positions.

[0047] For each fish target Based on the corresponding binary mask, a minimum bounding rectangle of the fish target region is generated, and then expanded by a certain factor (e.g., 1.2 times) to obtain a local rectangular safe box. The pixel value inside the rectangular safety frame is 1, and the pixel value outside the rectangular safety frame is 0.

[0048] Secure background area With rectangular safety frame Perform the Hadamard product operation to obtain the fish target. Local security background mask : .

[0049] Fish targets The local security background mask is the fish target. Local security background mask for the fish target area.

[0050] (5) Combine the local security background masks of all fish target areas into the local security background mask of the memory frame.

[0051] (iii) Pixel mask generation.

[0052] In this embodiment, a query frame is used as input. The query encoder determines the pixel features, and the pixel features are matched with the foreground memory and background memory respectively to obtain the foreground readout and background readout. The background readout is used to suppress the foreground readout to obtain the pixel readout. The pixel readout and the target memory are used as input, and the target encoder and decoder determine the pixel mask of the query frame.

[0053] The query encoder determines the pixel features of the query frame, and then performs similarity matching between these pixel features and foreground and background memories to obtain foreground and background readouts. Specifically, this involves calculating the negative weighted Euclidean distance between the pixel features of the query frame and the memory features (i.e., pixel features) of the foreground (or background) memory. The memory features are then sorted in descending order of negative weighted Euclidean distance, and the top-ranked features are selected as the preferred memory features. The Softmax function is used to normalize the negative weighted Euclidean distance of each preferred memory feature to obtain its attention weight. Based on these attention weights, all preferred memory features are summed in a weighted manner to obtain the foreground (or background) readout. The foreground and background readouts are high-dimensional feature representations that correspond one-to-one with the pixels in the query frame and incorporate historical memory information. Specifically, they refer to the historical foreground (or historical background) features most similar to each pixel in the query frame.

[0054] After obtaining the foreground and background readouts, background suppression attention is designed to suppress the foreground readouts, thus obtaining pixel readouts.

[0055] At this point, background readout is used to suppress foreground readout to obtain pixel readout, which specifically includes the following steps.

[0056] (1) Perform linear projection on the foreground readout to obtain the query vector, perform linear projection on the background readout to obtain the key vector and value vector, and calculate the interference readout based on the query vector, key vector and value vector.

[0057] When selectively suppressing foreground readout using background readout, a query vector is generated based on the foreground readout, and a key vector and value vector are generated based on the background readout. Subsequently, based on the query vector, key vector, and value vector, interference readouts are retrieved from the background memory. It represents the aggregation result of background information that is highly similar to the foreground and may cause confusion.

[0058] The query vector is encoded from the foreground readout, which is the product of the first linear projection matrix and the foreground readout. The key vector and value vector are encoded from the background readout, with the key vector being the product of the second linear projection matrix and the background readout, and the value vector being the product of the third linear projection matrix and the background readout.

[0059] The formula for calculating interference readout is: ; in, To interfere with readout; For query vector; The key vector; It is a value vector.

[0060] (2) Calculate the difference between the foreground readout and the interference readout to obtain the pixel readout.

[0061] By subtracting the distorting readout from the foreground readout, a clean pixel readout is obtained: ; in, Read out for pixels; Read out the foreground.

[0062] After obtaining the pixel readout, the pixel readout and the target memory are interacted bidirectionally. Specifically, the pixel readout and the target memory are used as inputs, and the pixel mask of the query frame is determined by the target encoder and decoder.

[0063] The target encoder can use the Transformer model, and the decoder can use the Res-Net model.

[0064] To further address the issues of target confusion and error accumulation, and improve segmentation accuracy, this embodiment performs uncertain region redistribution and small-area noise filtering (which is a small interference filtering) on ​​the pixel mask.

[0065] At this point, using pixel readout and target memory as input, the pixel mask of the query frame is determined by the target encoder and decoder. Specifically, this includes: using pixel readout and target memory as input, determining the initial pixel mask of the query frame by the target encoder and decoder, and performing uncertain region redistribution and small area noise filtering on the initial pixel mask to obtain the pixel mask of the query frame.

[0066] The process of redistributing uncertain regions and filtering small-area noise from the initial pixel mask to obtain the pixel mask of the query frame includes the following steps.

[0067] (1) Based on the initial pixel mask, determine the uncertain pixel points in the query frame. The probability difference between the two categories with the highest probability is less than the preset difference, and the probability of the two categories with the highest probability is greater than the preset probability of the pixel point.

[0068] For each pixel in the query frame, the initial pixel mask includes the probability that the pixel belongs to each category, and the category with the highest probability is the final category of the pixel.

[0069] Determine the uncertain pixels in the query frame based on the initial pixel mask. : ; in, The probability of the category with the highest probability; The probability of the category with the second highest probability; This is the preset difference; This is a preset probability. Pixels with a value of 1 are considered uncertain pixels.

[0070] (2) For all uncertain pixels, group the two uncertain pixels with the highest probability that belong to the same category into a group to obtain multiple groups.

[0071] Pixels are grouped according to the two categories with the highest probability of uncertain pixels. Specifically, uncertain pixels with the same probability in the two categories are grouped together. If uncertain pixels belong to the same group, it means that these uncertain pixels are fluctuating between the same two categories, resulting in multiple groups.

[0072] (3) For each group, the uncertain pixels in the group are clustered by the connected component analysis method to obtain multiple clusters.

[0073] Each group of pixels is clustered using connected component analysis. Specifically, uncertain pixels within the same group are clustered using connected component analysis. Uncertain pixels belonging to the same connected component form a cluster, and each cluster represents a continuous uncertain region.

[0074] (4) For each cluster, calculate the comprehensive confidence score of each of the two categories with the highest probability, and determine the target category of the uncertain pixel in the cluster as the category with the highest comprehensive confidence score.

[0075] For each cluster, a comprehensive confidence score is calculated for each of the two candidate categories (i.e., the two categories with the highest probabilities), which integrates the Intersection over Union (IOU), relative distance, and average probability. The cluster is then assigned to the category with the highest comprehensive confidence score, i.e., the target category of the uncertain pixels in the cluster. All were identified as the categories with the highest overall confidence scores.

[0076] The formula for calculating the comprehensive confidence score is: ; in, For category The overall confidence score; For clusters In category The average probability of the cluster is equal to the cluster The category of each uncertain pixel is the category. The average probability; For clusters and category as category The set of pixels The relative distance between them is equal to the cluster any pixel and set The ratio of the minimum distance between any two pixels to the length of the diagonal in the pixel mask; For clusters bounding box and category as category bounding box of the set of pixels The intersection-union ratio between them.

[0077] (5) The probability of uncertain pixels in each cluster is processed by the probability smoothing formula to obtain the intermediate pixel mask.

[0078] Enhance target categories using probability smoothing formulas The probability of this category is increased, while the probability of other categories is decreased.

[0079] The probability smoothing formula is: ; in, For pixels ( , The category is a category. The smoothed probability; This is the probability enhancement coefficient, which controls the magnitude of probability enhancement. For pixels ( , The category is a category. The probability of; The target category is the category with the highest overall confidence score in this case. The target region is the region corresponding to the cluster in this case.

[0080] After obtaining the smoothed probabilities, the Softmax function is used to ensure that the sum of the probabilities of each uncertain pixel belonging to each category is 1. At this point, the intermediate pixel mask can be obtained.

[0081] (1)-(5) form the uncertain region redistribution module.

[0082] (6) Determine the small area region in the intermediate pixel mask where the pixel area is within the preset area range, determine the category with the largest number of pixels among several pixels around the small area region, and determine the target category of all pixels in the small area region as the category with the largest number.

[0083] Based on the intermediate pixel mask, a small area region with a pixel area within the range [1, β] (i.e., a preset area range) is found. The pixel area refers to the number of pixels in a pixel region (pixel values ​​are the same and their positions are adjacent). β is one-eighth of the average pixel area of ​​each fish target in the first image frame. According to the category of the neighborhood composed of 20 pixels (not uncertain pixels) surrounding the small area region, the small area region is assigned to the category with the largest number of pixels in that neighborhood. That is, the target category of all pixels in the small area region is determined to be the category with the largest number. For example, if 18 out of 20 pixels are classified as background, 1 pixel is classified as a certain fish target, and 1 pixel is classified as another fish target, then the category with the largest number is background.

[0084] (7) The probability of each pixel in each small area is processed by the probability smoothing formula to obtain the pixel mask.

[0085] The probability smoothing formula at this point is the same as the probability smoothing formula in the uncertain region redistribution module. It only requires determining the target category as the most numerous category and the target region as a small area. After obtaining the smoothed probabilities, the Softmax function is used to ensure that the sum of the probabilities of each pixel belonging to each category is 1. At this point, the pixel mask can be obtained.

[0086] (6)-(7) form a small area noise filtering module.

[0087] To further address the target loss problem and improve segmentation accuracy, this embodiment uses local target re-verification to recover the fish target when it disappears.

[0088] At this point, if it is determined that there are missing fish targets based on the pixel mask, then after determining the pixel mask of the query frame, the semi-supervised fish video target segmentation method of this embodiment further includes: performing local target re-verification on the pixel mask to obtain an updated pixel mask, and using the updated pixel mask as the final pixel mask of the query frame.

[0089] The process of performing local target re-verification on the pixel mask to obtain the updated pixel mask includes the following steps.

[0090] (1) Record the missing fish targets as fish targets to be recovered, and determine the latest image frame containing fish targets to be recovered as the target frame.

[0091] The latest image frame refers to the image frame containing the fish target to be recovered that is located before and closest to the query frame.

[0092] (2) Determine the local target range based on the position of the fish target to be recovered in the target frame.

[0093] When a fish target is lost, the pixel mask of the target frame is used to determine the range of the fish target to be recovered. As the number of frames in which the fish target to be recovered has disappeared (the difference between the index of the query frame and the index of the target frame, where the index indicates which frame) increases, the local target range becomes 1.2 times the size of the previous frame for each additional frame. When the number of missing frames is greater than five, the local target range expands to the entire image frame; that is, if the number of missing frames is less than or equal to five, the local target range is 1.2. miss The range of the fish target to be recovered in the target frame, where miss is the number of frames that have disappeared. It needs to meet the constraint that it is located inside the image frame. If the number of frames that have disappeared is greater than 5, then the local target range is the entire image frame.

[0094] (3) Perform similarity matching between the local pixel features corresponding to the local target range in the query frame and the local pixel features corresponding to the local target range in each memory frame to obtain the similarity of each memory frame, and take the memory frames with the highest similarity as the target memory frames.

[0095] Pixel features include the features of each individual pixel. Therefore, the features of pixels located within the local target area constitute the local pixel features.

[0096] The similarity between the query frame and each memory frame in local pixel features is calculated using Euclidean distance. In this embodiment, the three memory frames with the highest similarity can be selected as target memory frames. Of course, other numbers can also be selected according to the needs, and the fused pixels can be read out afterward by calculating the target memory frames.

[0097] (4) The pixel readouts of all target memory frames are fused to obtain the fused pixel readouts.

[0098] When reading out pixels from the fused target memory frame, a time decay factor is introduced to reduce the impact of target memory frames with a long time distance on the query frame. The calculation formula for the fused pixel readout is as follows: ; ; ; in, For reading out the merged pixels; A set of target memory frames; For target memory frame The weights; For target memory frame Pixel readout; The weight of the first image frame is fixed at 1 because it has the highest reliability as a true annotation and does not decay over time. For target memory frames other than the first image frame The weights; For target memory frame Corresponding similarity; For target memory frame Corresponding similarity; The index for the query frame; For target memory frame The index.

[0099] (5) Using the fused pixel readout and target memory as input, the target encoder and decoder are used to determine the re-segmented pixel mask of the query frame.

[0100] The fused pixel readout and target memory are input together into the target encoder and decoder to obtain the re-segmented pixel mask, which determines the re-segmentation result of the local target range.

[0101] (6) The pixel mask and the re-segmented pixel mask are fused to obtain the updated pixel mask.

[0102] During fusion, for each pixel within the local target area, the target category of that pixel is determined based on the re-segmented pixel mask. Then, the probability of that pixel is processed using a probabilistic smoothing formula, the same as the one used in the uncertain region redistribution module. The only difference is that the target category is determined to be the pixel's target category, and the target region is defined as the local target area. After obtaining the smoothed probabilities, the Softmax function is used to ensure that the sum of the probabilities of each pixel belonging to each category is 1. At this point, the updated pixel mask is obtained, determining the final segmentation result of the query frame.

[0103] The above process constitutes the local target re-verification module.

[0104] The determination of the local target range is performed at the pixel feature scale to avoid spatial errors caused by downsampling.

[0105] This embodiment can segment the query frame based on a pixel mask, and mark the category of each pixel on the query frame. When segmenting, the size of the pixel mask needs to be mapped back to the size of the query frame. At this time, bilinear interpolation is used to maintain consistent spatial alignment.

[0106] (iv) Iterative loop.

[0107] Continue segmenting the next image frame, storing a memory frame every five frames (or other values) until all image frames are segmented. After obtaining the pixel masks of all image frames, overlay the pixel masks onto the image frames to restore the fish video to a segmented video, and output the segmented video.

[0108] In this embodiment, it determines whether the query frame is the last image frame to obtain a first determination result. If the first determination result is yes, the segmentation of all image frames is completed and the iteration ends. If the first determination result is no, it determines whether the number of image frames between the query frame and the previous memory frame has reached a preset number (the preset number is four when a memory frame is stored every five frames) to obtain a second determination result. If the second determination result is yes, the query frame is used as a memory frame, the next image frame of the query frame is used as the query frame of the next iteration, and the process returns to the step of "using all memory frames and the pixel mask of all memory frames as input, and using a memory encoder to determine the foreground memory, background memory, and target memory". If the second determination result is no, the next image frame of the query frame is used as the query frame of the next iteration, and the process returns to the step of "using the query frame as input, and using a query encoder to determine pixel features".

[0109] During training, this embodiment pre-trains on a collected static fish dataset (including images of static fish individuals and their labels, where the label refers to the category of each pixel), and then performs main training on the public datasets DAVIS 2017, YouTube-VOS, and the collected fish video dataset (including fish videos and their labels). The main training is first jointly trained on DAVIS 2017 and YouTube-VOS, and then separately trained on the fish video dataset. During the joint training phase, a joint sampling strategy is adopted, randomly selecting video sequences from the two datasets for end-to-end optimization.

[0110] The image frames in the input video were uniformly scaled to 480p. During training, the AdamW optimizer was used, employing a loss function that combines classification loss based on Soft Dice and contrastive loss based on Info-NCE, with equal weights for the classification and contrastive losses.

[0111] At this point, when training the memory encoder, query encoder, target encoder, and decoder, the loss used is a weighted sum of the classification loss and the contrastive loss. The classification loss uses the Soft Dice loss function, and the contrastive loss uses the Info-NCE loss function.

[0112] The formula for calculating classification loss is: ; in, For classification loss; Target number of fish; For pixels The category is a category The predicted probability; For pixels The category is a category The true probability.

[0113] The formula for calculating the contrast loss is: ; in, To compare the losses; Foreground-background feature contrast loss; This is a feature contrast loss between various fish targets in the foreground.

[0114] Foreground-background feature contrast loss is used to distinguish foreground and background features. First, foreground feature prototypes are extracted. and background features prototype For each Construct positive samples and negative samples Calculate the cross-entropy of the binary classification.

[0115] The formula for calculating the foreground-background feature contrast loss is: ; ; ; ; ; ; in, For the category of fish The number of pixels; The target space in the entire video, specifically targets categorized as fish. A collection of pixels; For batch Input image frame Fish targets The first loss value; For batch Input image frame Fish targets The first positive sample; For batch Input image frame Fish targets Negative samples; For batch Input image frame Fish targets Foreground feature prototype; For temperature parameters; For batch Input image frame Fish targets Background feature prototype; , indicating that for vector L2 normalization; It is the set of pixels in an image frame; For batch Input image frame Fish targets pixels The pixel mask information value (if it is a fish target) If the result is 1, then it is 1; otherwise, it is 0. (Based on batch) Input image frame The pixel mask is determined; For batch Input image frame Fish targets pixels The pixel readout information value (is the pixel point in the pixel readout) (values), based on batch Input image frame The pixel readout is determined; To prevent division by zero constants, >0; For batch Input image frame Fish targets pixels Background mask information (pixels in a local security background mask) (values), based on batch Input image frame The local security background mask is determined.

[0116] The feature contrast loss between various fish targets in the foreground is used to distinguish different target instances to minimize feature confusion. To ensure temporal continuity, positive and negative samples are constructed using features from the current image frame and the previous image frame, and the binary classification cross-entropy is calculated.

[0117] The formula for calculating the feature contrast loss between various fish targets in the foreground is: ; ; ; ; in, For batch Input image frame Fish targets The second loss value; For batch Input image frame Fish targets The second positive sample; Target number of fish; For batch Input image frame Fish targets and fish targets Negative samples; For batch Input image frame Fish targets Foreground feature prototype, image frame For image frames The previous image frame; For batch Input image frame Fish targets Foreground feature prototype; For batch Input image frame Fish targets The foreground features prototype.

[0118] This embodiment designs a foreground-background memory fusion module, which explicitly introduces background context information through a dual-path memory reading mechanism for both foreground and background. This overcomes the interference caused by traditional methods that rely solely on foreground memory. Simultaneously, a background-guided suppression strategy effectively suppresses erroneous responses caused by background noise, significantly improving feature discrimination and resolving target confusion. Based on contrast loss, this embodiment effectively narrows the distance between the target foreground features and their corresponding memory prototypes in the embedding space, while simultaneously increasing the distance between the target and the background or other targets, further enhancing category distinguishability and reducing the probability of target confusion. To alleviate target confusion, error accumulation, and target loss, this embodiment also introduces a three-stage post-processing strategy. This involves redistributing uncertain regions, removing noise points caused by environmental interference, and recovering lost targets through a local target re-verification mechanism, thereby effectively improving segmentation accuracy.

[0119] This application also provides an application scenario in which the aforementioned semi-supervised fish video target segmentation method is applied. Specifically, the semi-supervised fish video target segmentation method provided in this embodiment can be applied to a fish farming scenario. A fish farming scenario includes a video segmentation stage, a display stage, and a farming stage. The video segmentation stage is used to segment fish videos to obtain segmented videos; the display stage is used to show the segmented videos to users; and the farming stage is used for users to scientifically farm fish based on the segmented videos. The semi-supervised fish video target segmentation method provided in this embodiment belongs to the video segmentation stage.

[0120] Example 2.

[0121] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a semi-supervised fish video target segmentation method.

[0122] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the semi-supervised fish video target segmentation method of Embodiment 1.

[0124] Example 3.

[0125] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the semi-supervised fish video target segmentation method of Embodiment 1.

[0126] Example 4.

[0127] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the semi-supervised fish video target segmentation method of Embodiment 1.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of the relevant data are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0129] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A fish video target segmentation method based on semi-supervised, characterized in that, The semi-supervised fish video target segmentation method includes: Obtain a fish video and the pixel mask of the first image frame in the fish video; the fish video includes multiple image frames, and the pixel mask includes the category of each pixel in the image frame, the category including fish target and background; Use the first image frame as the memory frame and the second image frame as the query frame. Using all the memory frames and the pixel mask of all the memory frames as input, the foreground memory, background memory, and target memory are determined by the memory encoder; Using the query frame as input, a query encoder is used to determine pixel features. The pixel features are then matched with the foreground memory and the background memory for similarity to obtain foreground readout and background readout. The background readout is used to suppress the foreground readout to obtain pixel readout. Using the pixel readout and the target memory as input, a target encoder and decoder are used to determine the pixel mask of the query frame. Determine whether the query frame is the last image frame to obtain a first determination result; if the first determination result is no, determine whether the number of image frames between the query frame and the previous memory frame reaches a preset number to obtain a second determination result; if the second determination result is yes, use the query frame as a memory frame, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using all the memory frames and the pixel masks of all the memory frames as input, and using a memory encoder to determine the foreground memory, background memory, and target memory"; if the second determination result is no, use the next image frame of the query frame as the query frame of the next iteration, and return to the step of "using the query frame as input, and using a query encoder to determine pixel features".

2. The fish video object segmentation method based on semi-supervision according to claim 1, characterized in that, The foreground readout is suppressed by the background readout to obtain pixel readout, specifically including: A query vector is obtained by linearly projecting the foreground readout; The background readout is linearly projected to obtain the key vector and value vector; Based on the query vector, the key vector, and the value vector, the interference readout is calculated; Calculate the difference between the foreground readout and the interference readout to obtain the pixel readout; The calculation formula for the interference readout is as follows: ； wherein, is an interference readout; is a query vector; is a key vector; is a value vector. 3.The fish video object segmentation method based on semi-supervision according to claim 1, characterized in that, The foreground memory includes a memory frame and the pixel features, pixel readout, and pixel mask of the memory frame; the background memory includes a memory frame and the pixel features, pixel readout, and local safe background mask of the memory frame. The method for calculating the local security background mask includes: For each pixel in the memory frame, determine whether the category of the pixel is a fish target based on the pixel mask. If yes, set the pixel value of the pixel to a first value; otherwise, set the pixel value of the pixel to a second value to obtain the global foreground region mask. The foreground region in the global foreground region mask is dilated to obtain the dilated foreground region; the foreground region is a region composed of pixels with a first pixel value and adjacent positions. The region in the global foreground region mask other than the inflated foreground region is used as the safe background region; For each fish target region in the global foreground region mask, a minimum bounding rectangle of the fish target region is generated. The minimum bounding rectangle is enlarged to obtain a rectangular safe box. The safe background region is then subjected to a Hadamard product operation with the rectangular safe box to obtain a local safe background mask for the fish target region. The fish target region is a region composed of pixels of the same type and adjacent positions. The local security background masks of all the fish target regions are combined to form the local security background mask of the memory frame. 4.The fish video object segmentation method based on semi-supervision according to claim 1, characterized in that, Using the pixel readout and the target memory as input, the pixel mask of the query frame is determined by using a target encoder and a decoder. Specifically, this includes: using the pixel readout and the target memory as input, determining the initial pixel mask of the query frame by using a target encoder and a decoder, and performing uncertain region redistribution and small area noise filtering on the initial pixel mask to obtain the pixel mask of the query frame. Specifically, the pixel mask of the query frame is obtained by performing uncertain region redistribution and small-area noise filtering on the initial pixel mask, including: Based on the initial pixel mask, uncertain pixels in the query frame are determined; the uncertain pixels are pixels whose probability difference between the two highest probability categories is less than a preset difference, and whose probabilities of the two highest probability categories are both greater than a preset probability. For all the uncertain pixels, the two uncertain pixels with the highest probability that belong to the same category are grouped together to obtain multiple groups; For each group, the uncertain pixels in the group are clustered using the connected component analysis method to obtain multiple clusters; For each cluster, calculate the comprehensive confidence score for each of the two categories with the highest probability, and determine the target category of all uncertain pixels in the cluster as the category with the highest comprehensive confidence score; The probability of uncertain pixels in each cluster is processed using a probability smoothing formula to obtain an intermediate pixel mask; Determine a small area region in the intermediate pixel mask where the pixel area is within a preset area range, determine the category with the largest number of pixels among several pixels around the small area region, and determine the target category of all pixels in the small area region as the category with the largest number of pixels. The probability of each pixel in the small area is processed using a probability smoothing formula to obtain a pixel mask.

5. The fish video object segmentation method based on semi-supervision according to claim 4, characterized in that, The formula for calculating the comprehensive confidence score is as follows: ； in, For category The overall confidence score; For clusters In category The average probability of the cluster is equal to the cluster The category of each uncertain pixel is the category. The average probability; For clusters and category as category The set of pixels The relative distance between them; For clusters bounding box and category as category bounding box of the set of pixels The crossover and union ratio between them; The probability smoothing formula is as follows: ； wherein, is the class of pixel point ( , ) is the smoothed probability of class ; is the probability enhancement coefficient; is the probability of pixel point ( , ) being of class ; is the target class; is the target region, referring to a cluster corresponding region or a small area region.

6. The fish video object segmentation method based on semi-supervision according to claim 1, characterized in that, If it is determined that there is a missing fish target based on the pixel mask, then after determining the pixel mask of the query frame, the semi-supervised fish video target segmentation method further includes: performing local target re-verification on the pixel mask to obtain an updated pixel mask, and using the updated pixel mask as the final pixel mask of the query frame. Specifically, performing local target re-verification on the pixel mask to obtain the updated pixel mask includes: The missing fish targets are recorded as fish targets to be recovered, and the latest image frame containing the fish targets to be recovered is determined as the target frame; Based on the position of the fish target to be recovered in the target frame, the local target range is determined; The local pixel features corresponding to the local target range in the query frame are matched with the local pixel features corresponding to the local target range in each memory frame to obtain the similarity of each memory frame, and the memory frames with the highest similarity are taken as the target memory frames. The pixel readouts of all the target memory frames are fused to obtain the fused pixel readouts; Using the fused pixel readout and the target memory as input, the re-segmented pixel mask of the query frame is determined by the target encoder and decoder; The pixel mask and the re-segmented pixel mask are fused to obtain the updated pixel mask.

7. The fish video object segmentation method based on semi-supervision according to claim 6, characterized in that, The formula for calculating the readout of the fused pixels is: ； ； ； in, For reading out the merged pixels; A set of target memory frames; For target memory frame The weights; For target memory frame Pixel readout; The weights for the first image frame; For target memory frames other than the first image frame The weights; For target memory frame Corresponding similarity; For target memory frame Corresponding similarity; The index for the query frame; For target memory frame The index. 8.The fish video object segmentation method based on semi-supervision according to claim 1, characterized in that, When training the memory encoder, the query encoder, the target encoder, and the decoder, the loss used is a weighted sum of the classification loss and the contrastive loss. The classification loss adopts the Soft Dice loss function, and the contrastive loss adopts the Info-NCE loss function. The formula for calculating the contrast loss is: ； in, To compare the losses; Foreground-background feature contrast loss; This is a feature contrast loss between various fish targets in the foreground; The formula for calculating the foreground-background feature contrast loss is as follows: ； ； ； ； in, For the category of fish The number of pixels; For the category of fish A collection of pixels; For batch Input image frame Fish targets The first loss value; For batch Input image frame Fish targets The first positive sample; For batch Input image frame Fish targets Negative samples; For batch Input image frame Fish targets Foreground features prototype; For temperature parameters; For batch Input image frame Fish targets Background feature prototype; The formula for calculating the feature contrast loss between the various fish targets in the foreground is as follows: ； ； ； ； in, For batch Input image frame Fish targets The second loss value; For batch Input image frame Fish targets The second positive sample; Target number of fish; For batch Input image frame Fish targets and fish targets Negative samples; For batch Input image frame Fish targets Foreground feature prototype, image frame For image frames The previous image frame; For batch Input image frame Fish targets Foreground features prototype; For batch Input image frame Fish targets The foreground features of the prototype.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the semi-supervised fish video target segmentation method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the semi-supervised fish video target segmentation method according to any one of claims 1-8.

Images