School behavior recognition method, device, equipment and medium

Abstract

The application provides a fish school behavior recognition method, device and equipment and medium, and relates to the field of aquaculture, which comprises the following steps: all image frames in an initial image data set are segmented according to an encoder gradient to obtain first gradient features of a preset gradient; the remaining gradient features in each preset gradient are connected to a decoder to decode second gradient features of the preset gradient in each image frame; the first gradient features and the second gradient features in each image frame are fused to obtain fish school segmentation images of each image frame, and the initial image data set is traversed until a fish school segmentation image set is obtained; and fish school dispersion indexes and fish school activity indexes are calculated according to the fish school segmentation image set to obtain fish school behavior according to a long short-term memory network model. The application solves the problem that a target fish school region is similar to the background of an image region and cannot be distinguished, realizes accurate segmentation of the boundary of the fish school target, increases the recognition accuracy, reduces the parameter quantity and the calculation amount, and improves the efficiency of fish breeding.

Description

Technical Field

[0001] This invention relates to the field of aquaculture, and in particular to a method, apparatus, equipment and medium for recognizing fish school behavior. Background Technology

[0002] With the continuous development of aquaculture, the study of fish behavior is becoming increasingly important. Numerous studies have shown that behavior is a sensitive indicator of environmental change within an ecosystem or under controlled conditions. In intensive aquaculture, fish behavior is primarily influenced by temperature, oxygen levels, and external stimuli. Common fish behaviors encountered in actual aquaculture include feeding behavior, stress behavior, and swimming behavior. Real-time monitoring of fish swimming activity is crucial for improving aquaculture efficiency in order to identify these behaviors.

[0003] To accurately analyze fish behavior, behavioral information should be recorded in an objective, precise, and low-cost manner. Early behavioral studies relied primarily on direct observation, a method that depends on the observer's subjective judgment and is time-consuming and labor-intensive. Compared to human observation, telemetry and acoustic technologies, while offering the advantage of structural parameterization, are susceptible to water surface fluctuations. Furthermore, telemetry is an invasive technology that can damage fish and is unsuitable for intensive aquaculture scenarios.

[0004] However, monitoring and analyzing fish behavior in intensive aquaculture environments is a challenging task. Due to poor underwater lighting conditions, the quality of the acquired images is poor, and there are problems such as the similarity between the target and the background. Most model algorithms designed for underwater fish are difficult to meet real-time requirements. Summary of the Invention

[0005] This invention provides a method, device, equipment, and medium for fish school behavior recognition, which addresses the technical deficiencies in existing technologies that cannot achieve intelligent detection of abnormal fish behavior. This invention proposes a sensitive, non-invasive, and widely applicable technical solution for quantifying spatial changes in underwater fish schools and timely monitoring fish school behavior.

[0006] In a first aspect, the present invention provides a method for recognizing fish school behavior, comprising:

[0007] Based on the encoder in the fish swarm semantic segmentation model, gradient segmentation is performed on all image frames in the initial image dataset, and the first gradient feature of the preset gradient in each image frame is obtained.

[0008] The residual gradient features in each preset gradient are connected to the decoder according to the cross-stage connection method, so as to decode the second gradient features of the preset gradient in each image frame.

[0009] The first gradient feature and the second gradient feature in each image frame are fused to obtain the fish swarm segmentation image of each image frame. All initial image datasets are traversed until the fish swarm segmentation image set is obtained.

[0010] Based on the fish swarm segmentation image set, fish swarm dispersion index and fish swarm activity index are calculated to obtain fish swarm behavior according to the long short-term memory network model.

[0011] The fish group segmentation image set is a collection of images after segmenting the fish group and the background image;

[0012] The fish behavior includes feeding behavior, stress behavior, and normal swimming behavior.

[0013] According to a fish school behavior recognition method provided by the present invention, the gradient segmentation of all image frames in the initial image dataset, and the acquisition of the first gradient feature of a preset gradient in each image frame, includes:

[0014] Gradient segmentation is performed on any image frame in the initial image dataset to determine the initial information features and initial residual features in the upper gradient of the image frame.

[0015] The initial information features are downsampled to determine the next information feature and the next residual feature in the next layer gradient.

[0016] The next information feature is downsampled and, according to a preset gradient, until the underlying information feature and the remaining underlying feature in the underlying gradient are determined.

[0017] All information features from the initial information features to the bottom information features are determined as first gradient features, and all remaining features from the initial remaining features to the bottom remaining features are determined as remaining gradient features;

[0018] Iterate through all image frames until the first gradient feature of the preset gradient in each image frame is obtained.

[0019] According to a fish school behavior recognition method provided by the present invention, the step of connecting the residual gradient features in each preset gradient to the decoder according to a cross-stage connection method to decode the second gradient features of the preset gradient in each image frame includes:

[0020] Decode the residual gradient features in each gradient from the bottom gradient to the top gradient direction;

[0021] In the process of decoding the residual gradient features in each gradient, the first gradient feature in the previous gradient and the first gradient feature in the current gradient are channel-merged, and then the feature is added to the decoded residual gradient feature in the current gradient. The result is then input into the attention mechanism module, and the determined residual gradient feature in the current gradient is determined as the residual gradient feature in the next gradient, until the second gradient feature is determined.

[0022] The attention mechanism module includes at least a channel attention mechanism module and a spatial attention mechanism module.

[0023] According to a fish school behavior recognition method provided by the present invention, the step of fusing a first gradient feature and a second gradient feature in each image frame to obtain a fish school segmentation image for each image frame includes:

[0024] The initial residual features and the second gradient features are fused to obtain a fish swarm segmentation image for each image frame.

[0025] According to a fish school behavior recognition method provided by the present invention, the step of calculating fish school dispersion indicators and fish school activity indicators based on the fish school segmentation image set includes:

[0026] The fish dispersion index in each fish group segmentation image is determined based on the fish group region and the perimeter of each region in each fish group segmentation image. The fish group segmentation image set is traversed until the total fish group dispersion index is determined.

[0027] Fish activity indicators are determined based on the overlapping areas and time intervals of adjacent fish segmentation images in the fish segmentation image set.

[0028] According to the fish school behavior recognition method provided by the present invention, before obtaining the fish school behavior based on the long short-term memory network model, the method further includes:

[0029] Determine the fish behavior and fish index information for each sample video data based on a preset number of sample video data.

[0030] The sample video data of a predetermined number are divided into training set and test set according to a predetermined ratio in order to construct a long short-term memory network model.

[0031] The fish school indicator information includes at least fish school dispersion indicators and fish school activity indicators.

[0032] According to the fish school behavior recognition method provided by the present invention, before gradient segmentation of all image frames in the initial image dataset, the method further includes:

[0033] Image frames are extracted from the video clips of the fish schools to be identified in order to construct an initial image dataset.

[0034] Secondly, a fish school behavior recognition device is also provided, including:

[0035] Acquisition Unit: Used to segment all image frames in the initial image dataset according to the encoder in the fish swarm semantic segmentation model, and to acquire the first gradient feature of the preset gradient in each image frame;

[0036] Connection unit: used to connect the residual gradient features in each preset gradient to the decoder according to the cross-stage connection method, so as to decode the second gradient features of the preset gradient in each image frame;

[0037] Fusion unit: used to fuse the first gradient features and the second gradient features in each image frame to obtain the fish swarm segmentation image of each image frame, traversing all initial image datasets until the fish swarm segmentation image set is obtained;

[0038] Calculation unit: used to calculate fish dispersion index and fish activity index based on the fish segmentation image set, so as to obtain fish behavior according to the long short-term memory network model;

[0039] The fish group segmentation image set is a collection of images after segmenting the fish group and the background image;

[0040] The fish behavior includes feeding behavior, stress behavior, and normal swimming behavior.

[0041] Thirdly, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fish school behavior recognition method.

[0042] Fourthly, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the fish swarm behavior recognition method.

[0043] This invention provides a method, apparatus, device, and medium for fish school behavior recognition, used for real-time monitoring and recognition of underwater fish school behavior in factory farming. It solves the problems of low segmentation accuracy and training redundancy in existing models. By improving the semantic segmentation model and adopting a cross-stage connection method, it solves the technical problem that the target fish school region in the underwater fish school dataset is similar to the background of the image region and cannot be effectively distinguished. Thus, it can achieve accurate segmentation of the boundary of the fish school target in the image. While increasing the recognition accuracy, it reduces the number of parameters and the amount of computation, and improves the efficiency of fish farming in a lossless, accurate, and easy-to-deploy manner. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is one of the flowcharts of the fish school behavior recognition method provided by the present invention;

[0046] Figure 2 This is a schematic diagram of the process for obtaining the first gradient feature of a preset gradient in each image frame provided by the present invention;

[0047] Figure 3 This is a flowchart illustrating the calculation of fish dispersion and fish activity indicators provided by the present invention.

[0048] Figure 4 This is the second flowchart of the fish school behavior recognition method provided by the present invention;

[0049] Figure 5 This is one of the structural schematic diagrams of the fish school behavior recognition method provided by the present invention;

[0050] Figure 6 This is the second structural schematic diagram of the fish school behavior recognition method provided by the present invention;

[0051] Figure 7 This is the third structural schematic diagram of the fish school behavior recognition method provided by the present invention;

[0052] Figure 8 This is the fourth structural schematic diagram of the fish school behavior recognition method provided by the present invention;

[0053] Figure 9 This is the fifth structural schematic diagram of the fish school behavior recognition method provided by the present invention;

[0054] Figure 10 This is a schematic diagram of the structure of the fish school behavior recognition device provided by the present invention;

[0055] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] With the continuous development of computer vision, intelligent detection of abnormal fish behavior has become an inevitable trend. It improves fish farming efficiency and achieves sustainable development in a non-destructive, accurate, and easy-to-deploy manner. Therefore, using intelligent technology to detect fish behavior is of great significance for ensuring fish welfare and guaranteeing industrial and ecological benefits.

[0058] Figure 1 This is one of the flowcharts illustrating the fish school behavior recognition method provided by the present invention. The present invention provides a fish school behavior recognition method, comprising:

[0059] Based on the encoder in the fish swarm semantic segmentation model, gradient segmentation is performed on all image frames in the initial image dataset, and the first gradient feature of the preset gradient in each image frame is obtained.

[0060] The residual gradient features in each preset gradient are connected to the decoder according to the cross-stage connection method, so as to decode the second gradient features of the preset gradient in each image frame.

[0061] The first gradient feature and the second gradient feature in each image frame are fused to obtain the fish swarm segmentation image of each image frame. All initial image datasets are traversed until the fish swarm segmentation image set is obtained.

[0062] Based on the fish swarm segmentation image set, fish swarm dispersion index and fish swarm activity index are calculated to obtain fish swarm behavior according to the long short-term memory network model.

[0063] The fish group segmentation image set is a collection of images after segmenting the fish group and the background image;

[0064] The fish behavior includes feeding behavior, stress behavior, and normal swimming behavior.

[0065] In step 101, those skilled in the art will understand that the present invention improves the semantic segmentation model for segmentation of underwater fish school images, namely, the fish school semantic segmentation model. Further, all image frames from the initial image dataset are input into the fish school semantic segmentation model for underwater fish school behavior recognition. The fish school semantic segmentation model disclosed in this invention mainly includes an improved lightweight encoder-decoder framework, a redesigned cross-stage connection method, and a feature fusion module.

[0066] like Figure 6 As shown, Figure 6 This is the second structural schematic diagram of the fish swarm behavior recognition method provided by this invention. It shows the structural framework of the encoder. The encoder in the lightweight encoder-decoder mainly adopts the idea of ​​segmentation and fusion, which can achieve richer gradient combinations while reducing the amount of computation. It divides the feature map of the base layer into two parts, and then merges the features generated from each part. This method can realize the segmentation of the gradient flow, allowing the gradient flow to propagate in two branch paths. The propagated gradient information may have different correlations. The merging method helps to enhance the learning ability of the network. Therefore, the improved encoder can effectively reduce the amount of computation and improve the inference speed and accuracy.

[0067] The preset gradient can be configured according to the number of encoders or the preset of the segmented gradient stream. Each gradient will be divided into two parts: one part is the first gradient feature and the other part is the remaining gradient feature. In step 101, it is mainly used to obtain the first gradient feature in each gradient in each image frame.

[0068] In step 102, the residual gradient features in each preset gradient are connected to the decoder according to the cross-stage connection method to decode the second gradient features of the preset gradient in each image frame, such as... Figure 7 As shown, Figure 7 This is the third structural schematic diagram of the fish school behavior recognition method provided by the present invention. It shows the structural framework of the decoder. The decoder uses bilinear interpolation to upsample, which avoids generating redundant noise while expanding the feature map.

[0069] This invention employs a cross-stage connection method, such as... Figure 5 As shown, Figure 5 This is one of the structural schematic diagrams of the fish school behavior recognition method provided by the present invention. The present invention trains an image frame using a semantic segmentation model to obtain an improved fish school semantic segmentation model. The improved fish school semantic segmentation model is then used to segment the fish school target in the image frame. Figure 5 This is an illustration of model improvement. Specifically, the encoder on the left learns the gradient features in each image frame and represents them as a gradient segmentation form that connects the gradient features from top to bottom across stages. The residual gradient features in each preset gradient are connected to the decoder on the right. The encoding process is a process in which the feature map gradually becomes smaller, while the decoding process is a process in which the feature map gradually becomes larger. The cross-stage connection method includes the encoding process. In the decoding process, decoding is performed from bottom to top. The residual gradient features of the lowest gradient are decoded, and through operations such as channel merging and feature addition with other residual gradient features in the horizontal direction of each preset gradient, the upper layer is decoded step by step to finally determine the second gradient feature.

[0070] In step 103, the first gradient feature and the second gradient feature in each image frame are fused to obtain the fish group segmentation image of each image frame. All initial image datasets are traversed until the fish group segmentation image set is obtained. Steps 101 to 102 of this invention process a single image frame, while in this invention, an initial image dataset composed of multiple image frames is used as input for subsequent determination of fish group behavior. Therefore, for each image frame, the operations in steps 101 to 102 need to be performed to determine multiple fused fish group segmentation images, and all fused fish group segmentation images are used as the fish group segmentation image set.

[0071] In step 104, based on the fish school segmentation image set, fish school dispersion index and fish school activity index are calculated to obtain fish school behavior according to the long short-term memory network model. The fish school segmentation image set is a collection of images after segmenting the fish school and the background image. Compared with the existing technology, which cannot solve the problem of underwater fish school targets and background similarity, it has higher recognition accuracy. Therefore, using the fish school segmentation image set obtained after step 103 to determine the fish school dispersion index and fish school activity index will greatly improve the accuracy of fish school behavior recognition. The fish school behavior includes feeding behavior, stress behavior and normal swimming behavior. Those skilled in the art will understand that the present invention aims to determine two indicators characterizing fish school behavior, namely the fish school dispersion index and the fish school activity index. The fish school dispersion index is used to indicate the dispersion of fish in the factory farming pond, while the fish school activity index is used to estimate the activity level of the fish school.

[0072] Optionally, before gradient segmentation of all image frames in the initial image dataset, the following steps are also included:

[0073] Image frames are extracted from the video clips of the fish schools to be identified in order to construct an initial image dataset.

[0074] In an optional embodiment, the present invention can use an underwater camera to collect video data of fish feeding in an intensive aquaculture workshop, then use computer programming language code to extract image frames from the video data, and use all image frames as a dataset. The dataset is then labeled to include foreground pixels (underwater fish) and background pixels. Optionally, the dataset size is 150 images, of which 100 images are used for the training set and 50 images are used for the test set. The semantic segmentation model is trained based on the training set and the test set to determine the improved fish swarm semantic segmentation model.

[0075] After determining the improved fish swarm semantic segmentation model, video data of the fish swarm before and after feeding was collected again according to the above operation method, and image frames were extracted from the video segments of the fish swarm to be identified in order to construct the initial image dataset.

[0076] This invention can be used for real-time monitoring and identification of underwater fish behavior in intensive recirculating aquaculture systems. It first uses an underwater camera to collect video datasets of fish in an intensive recirculating aquaculture workshop to construct an initial image dataset. Then, after labeling, the initial image dataset is fed into a semantic segmentation model to accurately extract fish features from the image frames for identification. The improved semantic segmentation model has fewer parameters and less computation, and can meet real-time requirements.

[0077] This invention also proposes a lightweight encoder-decoder framework, which effectively reduces the complexity of the model while accurately extracting underwater fish features. Furthermore, this invention redesigns a cross-stage connection method, which efficiently improves the model's accuracy and helps solve the problem of underwater targets being similar to the background. Finally, the added feature fusion module fuses the model's shallow and deep features, further enriching the features and increasing the model's accuracy.

[0078] This invention also utilizes an improved fish swarm semantic segmentation model to identify and calculate a set of segmented fish images, thereby obtaining two indicators characterizing fish swarm behavior. These two indicators quantify the spatial distribution and activity level of the fish swarm. Testing shows that these two indicators effectively reflect spatial changes within the group. Inputting the indicator information into a trained Long Short-Term Memory (LSTM) model enables the classification and identification of fish swarm behavior.

[0079] This invention provides a method, apparatus, device, and medium for fish school behavior recognition, used for real-time monitoring and recognition of underwater fish school behavior in factory farming. It solves the problems of low segmentation accuracy and training redundancy in existing models. By improving the semantic segmentation model and adopting a cross-stage connection method, it solves the technical problem that the target fish school region in the underwater fish school dataset is similar to the background of the image region and cannot be effectively distinguished. Thus, it can achieve accurate segmentation of the boundary of the fish school target in the image. While increasing the recognition accuracy, it reduces the number of parameters and the amount of computation, and improves the efficiency of fish farming in a lossless, accurate, and easy-to-deploy manner.

[0080] Figure 2 This is a schematic diagram of the process for obtaining the first gradient feature of a preset gradient in each image frame provided by the present invention. The gradient segmentation of all image frames in the initial image dataset and the acquisition of the first gradient feature of the preset gradient in each image frame include:

[0081] Gradient segmentation is performed on any image frame in the initial image dataset to determine the initial information features and initial residual features in the upper gradient of the image frame.

[0082] The initial information features are downsampled to determine the next information feature and the next residual feature in the next layer gradient.

[0083] The next information feature is downsampled and, according to a preset gradient, until the underlying information feature and the remaining underlying feature in the underlying gradient are determined.

[0084] All information features from the initial information features to the bottom information features are determined as first gradient features, and all remaining features from the initial remaining features to the bottom remaining features are determined as remaining gradient features;

[0085] Iterate through all image frames until the first gradient feature of the preset gradient in each image frame is obtained.

[0086] In step 1011, gradient segmentation is performed on any image frame in the initial image dataset to determine the initial information features and initial residual features in the upper-level gradient of the image frame, such as... Figure 5 As shown, the image frame is gradient segmented into initial residual features in the horizontal direction and next information features in the vertical direction.

[0087] In step 1012, the initial information features are downsampled to determine the next information feature and the next residual feature in the next layer gradient; as follows Figure 5 As shown, the initial information features are downsampled by the encoder and then further segmented into the next information features and the next residual features in the next gradient.

[0088] In step 1013, combining steps 1011 and 1012 and following the above gradient segmentation rules, the next information feature is downsampled, and the underlying information feature and the remaining underlying feature are determined according to the preset gradient.

[0089] In step 1014, all image frames are traversed until the first gradient feature of the preset gradient in each image frame is obtained. Steps 1011 to 1013 are gradient segmentation processes performed on any one of the image frames. In step 1014, steps 1011 to 1013 are performed on all image frames until the first gradient feature of the preset gradient in each image frame is obtained.

[0090] Optionally, the step of connecting the residual gradient features in each preset gradient to the decoder according to the cross-stage connection method to decode the second gradient features of the preset gradient in each image frame includes:

[0091] Decode the residual gradient features in each gradient from the bottom gradient to the top gradient direction;

[0092] In the process of decoding the residual gradient features in each gradient, the first gradient feature in the previous gradient and the first gradient feature in the current gradient are channel-merged, and then the feature is added to the decoded residual gradient feature in the current gradient. The result is then input into the attention mechanism module, and the determined residual gradient feature in the current gradient is determined as the residual gradient feature in the next gradient, until the second gradient feature is determined.

[0093] The attention mechanism module includes at least a channel attention mechanism module and a spatial attention mechanism module.

[0094] Those skilled in the art will understand that the cross-stage connection method used in this invention mainly involves fusing the encoder with the shallow encoder, adding the fused feature map with the decoded feature map, and then passing the result through a merged channel-spatial attention mechanism to enhance the spatial-channel features and reduce noise. Finally, the encoder module generates the final feature map to prepare for subsequent steps.

[0095] The cross-stage connection method is shown in the following formula:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Among them, in equations (1) to (6) above, This represents the feature map output after passing through i encoder modules; It is the feature map corresponding to the encoder output; C n (·) represents an n×n convolution operation; D(·) represents downsampling; Concate represents channel merging and concatenation; Add represents feature map addition; Attention represents feeding features into the attention mechanism module; Decoder represents feeding the feature map into the decoder; express The result after processing express and The result after channel merging; express The result after processing; express The result of adding the feature maps of the two.

[0103] like Figure 8 As shown, Figure 8 This is the fourth structural schematic diagram of the fish school behavior recognition method provided by the present invention. In an optional embodiment, if It is a encoder of the same scale. If the output is , then the number of channels controlled by the 1×1 convolution kernel is 32. Low-level semantic information from a larger-scale encoder is conveyed by applying downsampling and a 3×3 convolutional kernel. Connect these features and map them to To enable the semantic segmentation model to learn better feature information, a merged channel-space attention mechanism is added to optimize the feature map.

[0104] The attention mechanism module includes at least a channel attention mechanism module and a spatial attention mechanism module. The channel attention mechanism module is used to compress spatial features. For example, if the input feature map is W×H×C, it is first compressed into 1×1×C, followed by an activation operation, which simulates the correlation between channels through two fully connected layers. The channel attention mechanism module can learn to use global information to selectively emphasize the expression of important information between channels while suppressing noise interference. The spatial attention mechanism module performs a compression operation on the feature map along the channels. The input feature map is compressed to generate a projection tensor W×H, and its activation operation is performed spatially. The channel attention mechanism module is beneficial for fine-grained image segmentation.

[0105] Optionally, the channel attention mechanism module is a spatial compression and channel excitation block. For any given feature map, spatial information is squeezed through a global average pooling layer, which can solve the channel dependency problem. In order to utilize the information summarized in the compression operation, the channel attention mechanism module can enhance the correlation between channels, thus improving the network's sensitivity to useful information. The spatial attention mechanism module is a channel compression and spatial excitation block that squeezes the feature map along the channels and excites it in space. The spatial attention mechanism module is beneficial for fine-grained image segmentation. Finally, the merged spatial and channel attention mechanism modules recalibrate the feature maps along the space and channels respectively, and output them after combination.

[0106] Optionally, fusing the first gradient features and the second gradient features in each image frame to obtain a fish swarm segmentation image for each image frame includes:

[0107] The initial residual features and the second gradient features are fused to obtain a fish swarm segmentation image for each image frame.

[0108] Figure 9 This is the fifth structural schematic diagram of the fish school behavior recognition method provided by the present invention, as shown below. Figure 9 As shown, the present invention aims to fuse shallow and deep features from a semantic segmentation model, then compress the spatial features through the attention mechanism module, followed by an activation operation. By simulating the correlation between channels through two fully connected layers, it can learn to selectively emphasize the expression of important information using global information. The features output by the attention mechanism module are fed into the final module for classification tasks.

[0109] Figure 3 This is a flowchart illustrating the calculation of fish school dispersion and activity indicators provided by the present invention. The step of calculating the fish school dispersion and activity indicators based on the fish school segmentation image set includes:

[0110] The fish dispersion index in each fish group segmentation image is determined based on the fish group region and the perimeter of each region in each fish group segmentation image. The fish group segmentation image set is traversed until the total fish group dispersion index is determined.

[0111] Fish activity indicators are determined based on the overlapping areas and time intervals of adjacent fish segmentation images in the fish segmentation image set.

[0112] In step 1041, the fish dispersion index in each fish segmentation image is determined based on the region of the fish in each fish segmentation image and the perimeter of each region. The fish segmentation image set is traversed until the total fish dispersion index is determined. The present invention calculates the index using the following formula:

[0113]

[0114] The calculation method involves summing the perimeters of the fish groups segmented by the model. In equation (7), a fish group may consist of regions formed by overlapping fish or may consist of only one region. The index is higher when the fish group is spreading and lower when it is clustered. Here, n represents the number of regions formed by one or more fish in an image; C i C represents the total perimeter of a school of fish in a given area; total This represents the total perimeter of all regions.

[0115] In step 1042, based on the overlapping area and time interval of adjacent fish group segmentation images in the fish group segmentation image set, the fish group activity index is determined. This invention calculates this index using the following formula:

[0116]

[0117] The calculation method estimates the area by subtracting the region that overlaps with the previous frame in chronological order from the current frame image. The area of ​​the remaining region is proportional to the movement of the fish school within a given time period, and can be used to estimate the swimming activity of the fish school. When the fish school is active, the calculation exponent is correspondingly high. In equation (8), A t dt represents the area of ​​image t after it has been processed by a semantic segmentation model; dt represents the time interval between the two images; A diff It can reflect the proportion of each movement of the fish school; A total Representing A t+dt and A t The sum of the two.

[0118] Figure 4 This is a second flowchart illustrating the fish school behavior recognition method provided by the present invention. Before obtaining the fish school behavior based on the Long Short-Term Memory network model, the method further includes:

[0119] Determine the fish behavior and fish index information for each sample video data based on a preset number of sample video data.

[0120] The sample video data of a predetermined number are divided into training set and test set according to a predetermined ratio in order to construct a long short-term memory network model.

[0121] The fish school indicator information includes at least fish school dispersion indicators and fish school activity indicators.

[0122] In step 201, the present invention first acquires video data including different fish school behaviors to use as a preset number of sample video data, and then extracts the indicator information of the sample video data to determine the fish school behavior and fish school indicator information of each sample video data.

[0123] In step 202, before the model training phase, optionally, the image size is scaled to a preset size, for example, 352×480. Further, the present invention uses the sample information of a first preset ratio as the training set to train the long short-term memory model; uses the sample information of a second preset ratio as the test set to verify the model, and optimizes the long short-term memory network model based on the verification results, thereby enabling the classification of fish behavior based on the input fish dispersion index and fish activity index, including feeding behavior, stress behavior and normal swimming behavior.

[0124] To more effectively demonstrate the beneficial effects of this invention, experiments were conducted on the collected dataset and compared with other models. Intersection over Union (IOU) and similarity coefficient (Dice) were used as accuracy evaluation indicators; parameter quantity and computational cost, frames per second (FPS) and inference time were used as lightweight model indicators. The experimental data was collected from an experimental pond in an intensive recirculating aquaculture workshop, with a diameter of 3.3 meters, a height of 64 cm, and a water depth of 55 cm. A camera was installed 15 cm from the bottom of the pond, and the camera acquired images at a rate of 15 frames per second. Finally, the video images were transmitted to the computer.

[0125] On the collected dataset, the present invention conducted ablation experiments, as shown in Table 1:

[0126]

[0127]

[0128] Table 1

[0129] On the collected dataset, the present invention was compared with existing technologies in an experiment, as shown in Table 2:

[0130]

[0131]

[0132] Table 2

[0133] Figure 10 This is a schematic diagram of the fish school behavior recognition device provided by the present invention. The present invention discloses a fish school behavior recognition device, including an acquisition unit 1: used to perform gradient segmentation of all image frames in the initial image dataset according to the encoder in the fish school semantic segmentation model, and to acquire the first gradient feature of the preset gradient in each image frame. The acquisition unit 1 can refer to the aforementioned step 101, which will not be repeated here.

[0134] The fish behavior recognition device further includes a connection unit 2: used to connect the remaining gradient features in each preset gradient to the decoder according to the cross-stage connection method, so as to decode the second gradient features of the preset gradient in each image frame. The connection unit 2 can refer to the aforementioned step 102, and will not be described in detail here.

[0135] The fish school behavior recognition device further includes a fusion unit 3: used to fuse the first gradient feature and the second gradient feature in each image frame to obtain the fish school segmentation image of each image frame, traversing all initial image datasets until the fish school segmentation image set is obtained. The fusion unit 3 can refer to the aforementioned step 103, and will not be described in detail here.

[0136] The fish school behavior recognition device further includes a computing unit 4: used to calculate fish school dispersion index and fish school activity index based on the fish school segmentation image set, so as to obtain fish school behavior according to the long short-term memory network model. The computing unit 4 can refer to the aforementioned step 104, and will not be described in detail here.

[0137] The fish group segmentation image set is a collection of images after segmenting the fish group and the background image;

[0138] The fish behavior includes feeding behavior, stress behavior, and normal swimming behavior.

[0139] This invention provides a method, apparatus, device, and medium for fish school behavior recognition, used for real-time monitoring and recognition of underwater fish school behavior in factory farming. It solves the problems of low segmentation accuracy and training redundancy in existing models. By improving the semantic segmentation model and adopting a cross-stage connection method, it solves the technical problem that the target fish school region in the underwater fish school dataset is similar to the background of the image region and cannot be effectively distinguished. Thus, it can achieve accurate segmentation of the boundary of the fish school target in the image. While increasing the recognition accuracy, it reduces the number of parameters and the amount of computation, and improves the efficiency of fish farming in a lossless, accurate, and easy-to-deploy manner.

[0140] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11As shown, the electronic device may include: a processor 110, a communication interface 120, a memory 130, and a communication bus 140, wherein the processor 110, the communication interface 120, and the memory 130 communicate with each other through the communication bus 140. The processor 110 can call logical instructions in the memory 130 to execute a fish school behavior recognition method. This method includes: gradient segmentation of all image frames in the initial image dataset according to the encoder in the fish school semantic segmentation model, obtaining the first gradient feature of a preset gradient in each image frame; connecting the remaining gradient features in each preset gradient to the decoder according to a cross-stage connection method to decode the second gradient feature of the preset gradient in each image frame; fusing the first gradient feature and the second gradient feature in each image frame to obtain a fish school segmentation image for each image frame; traversing all initial image datasets until a fish school segmentation image set is obtained; calculating fish school dispersion indicators and fish school activity indicators based on the fish school segmentation image set to obtain fish school behavior according to a long short-term memory network model; the fish school segmentation image set is a collection of images after segmenting the fish school from the background image; the fish school behavior includes feeding behavior, stress behavior, and normal swimming behavior.

[0141] Furthermore, the logical instructions in the aforementioned memory 130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a fish school behavior recognition method provided by the above methods. The method includes: gradient segmenting all image frames in an initial image dataset according to the encoder in a fish school semantic segmentation model, and obtaining a first gradient feature of a preset gradient in each image frame; connecting the remaining gradient features in each preset gradient to a decoder according to a cross-stage connection method to decode a second gradient feature of the preset gradient in each image frame; fusing the first gradient feature and the second gradient feature in each image frame to obtain a fish school segmentation image for each image frame; traversing all initial image datasets until a fish school segmentation image set is obtained; calculating a fish school dispersion index and a fish school activity index according to the fish school segmentation image set to obtain fish school behavior according to a long short-term memory network model; the fish school segmentation image set is a collection of images after segmenting the fish school and the background image; the fish school behavior includes feeding behavior, stress behavior, and normal swimming behavior.

[0143] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described methods to provide a method for recognizing fish school behavior. This method includes: gradient segmenting all image frames in an initial image dataset according to an encoder in a fish school semantic segmentation model, obtaining a first gradient feature of a preset gradient in each image frame; connecting the remaining gradient features in each preset gradient to a decoder according to a cross-stage connection method to decode a second gradient feature of the preset gradient in each image frame; fusing the first gradient feature and the second gradient feature in each image frame to obtain a fish school segmentation image for each image frame; traversing all initial image datasets until a fish school segmentation image set is obtained; calculating a fish school dispersion index and a fish school activity index based on the fish school segmentation image set to obtain fish school behavior according to a long short-term memory network model; the fish school segmentation image set is a collection of images after segmenting the fish school from the background image; the fish school behavior includes feeding behavior, stress behavior, and normal swimming behavior.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing fish school behavior, characterized in that, include: Based on the encoder in the fish swarm semantic segmentation model, gradient segmentation is performed on all image frames in the initial image dataset, and the first gradient feature of the preset gradient in each image frame is obtained. The residual gradient features in each preset gradient are connected to the decoder according to the cross-stage connection method, so as to decode the second gradient features of the preset gradient in each image frame. The first gradient feature and the second gradient feature in each image frame are fused to obtain the fish swarm segmentation image of each image frame. All initial image datasets are traversed until the fish swarm segmentation image set is obtained. Based on the fish swarm segmentation image set, fish swarm dispersion index and fish swarm activity index are calculated to obtain fish swarm behavior according to the long short-term memory network model. The fish group segmentation image set is a collection of images after segmenting the fish group and the background image; The fish behavior includes feeding behavior, stress behavior, and normal swimming behavior; The gradient segmentation process involves all image frames in the initial image dataset, obtaining the first gradient feature of a preset gradient in each image frame, including: Gradient segmentation is performed on any image frame in the initial image dataset to determine the initial information features and initial residual features in the upper gradient of the image frame. The initial information features are downsampled to determine the next information feature and the next residual feature in the next layer gradient. The next information feature is downsampled and, according to a preset gradient, until the underlying information feature and the remaining underlying feature in the underlying gradient are determined. All information features from the initial information features to the bottom information features are determined as first gradient features, and all remaining features from the initial remaining features to the bottom remaining features are determined as remaining gradient features; Traverse all image frames until the first gradient feature of the preset gradient in each image frame is obtained; the step of connecting the remaining gradient features in each preset gradient to the decoder according to the cross-stage connection method to decode the second gradient feature of the preset gradient in each image frame includes: Decode the residual gradient features in each gradient from the bottom gradient to the top gradient direction; In the process of decoding the residual gradient features in each gradient, the first gradient feature in the previous gradient and the first gradient feature in the current gradient are channel-merged, and then the feature is added to the decoded residual gradient feature in the current gradient. The result is then input into the attention mechanism module, and the determined residual gradient feature in the current gradient is determined as the residual gradient feature in the next gradient, until the second gradient feature is determined. The attention mechanism module includes at least a channel attention mechanism module and a spatial attention mechanism module.

2. The fish school behavior recognition method according to claim 1, characterized in that, The process of fusing the first gradient features and the second gradient features in each image frame to obtain a fish swarm segmentation image for each image frame includes: The initial residual features and the second gradient features are fused to obtain a fish swarm segmentation image for each image frame.

3. The fish school behavior recognition method according to claim 1, characterized in that, The step of calculating fish dispersion indicators and fish activity indicators based on the fish segmentation image set includes: The fish dispersion index in each fish group segmentation image is determined based on the fish group region and the perimeter of each region in each fish group segmentation image. The fish group segmentation image set is traversed until the total fish group dispersion index is determined. Fish activity indicators are determined based on the overlapping areas and time intervals of adjacent fish segmentation images in the fish segmentation image set.

4. The fish school behavior recognition method according to claim 1, characterized in that, Before obtaining the fish school behavior based on the Long Short-Term Memory network model, the following is also included: Determine the fish behavior and fish index information for each sample video data based on a preset number of sample video data. The sample video data of a predetermined number are divided into training set and test set according to a predetermined ratio in order to construct a long short-term memory network model. The fish school indicator information includes at least fish school dispersion indicators and fish school activity indicators.

5. The fish school behavior recognition method according to claim 1, characterized in that, Before gradient segmentation of all image frames in the initial image dataset, the following is also included: Image frames are extracted from the video clips of the fish schools to be identified in order to construct an initial image dataset.

6. A fish school behavior recognition device, employing the fish school behavior recognition method according to claim 1, characterized in that, include: Acquisition Unit: Used to segment all image frames in the initial image dataset according to the encoder in the fish swarm semantic segmentation model, and to acquire the first gradient feature of the preset gradient in each image frame; Connection unit: used to connect the residual gradient features in each preset gradient to the decoder according to the cross-stage connection method, so as to decode the second gradient features of the preset gradient in each image frame; Fusion unit: used to fuse the first gradient features and the second gradient features in each image frame to obtain the fish swarm segmentation image of each image frame, traversing all initial image datasets until the fish swarm segmentation image set is obtained; Calculation unit: used to calculate fish dispersion index and fish activity index based on the fish segmentation image set, so as to obtain fish behavior according to the long short-term memory network model; The fish group segmentation image set is a collection of images after segmenting the fish group and the background image; The fish behavior includes feeding behavior, stress behavior, and normal swimming behavior.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the fish school behavior recognition method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the fish school behavior recognition method as described in any one of claims 1 to 5.

Images