Intelligent feeding system for cultured fishes based on multi-feature fusion and control method of intelligent feeding system

Abstract

The invention belongs to the technical field of computer vision and deep learning, and relates to a cultured fish intelligent feeding system based on multi-feature fusion and a control method thereof, and the system comprises an image acquisition module, a space-motion feature extraction module, a motion track feature extraction module, a feeding state recognition module, a feeding control module and a feedback adjustment module. Extracting a depth optical flow image reflecting fish school feeding state characteristics through a fish school real-time video stream; classifying the fish school feeding state image samples to form a space-motion optical flow feature map data set; meanwhile, fish swimming trails are extracted, and a swimming trail feature map data set is formed; and training is carried out to obtain an optimal model weight, real-time identification is realized, an identification result is transmitted to the feeding control module, the feeding state of a fish school after feeding is completed is evaluated, the feeding amount is adjusted, or a feeding decision is optimized. According to the invention, real-time, accurate and efficient identification and decision control of the feeding state of the fish school can be realized, and intelligent feeding is realized on a low-cost edge end computing platform.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and deep learning, and relates to an intelligent feeding system and method for aquaculture, in particular to an intelligent feeding system for cultured fish based on multi-feature fusion and a control method thereof. BACKGROUND

[0002] With the rapid development of China's aquaculture industry, the demand for aquatic products continues to rise, and the scale of aquaculture is also expanding. At present, the feeding method in China's aquaculture is mainly artificial feeding, and the cost of feed accounts for more than 50% of the total cost. Since the artificial feeding method mainly depends on the experience of aquaculture personnel, it is easy to cause insufficient or excessive feeding, thereby causing feed waste, water pollution, and even affecting the quality of the cultured products. Under this background, intelligent feeding technology for aquaculture has emerged as the times require, and the adoption of scientific and intelligent aquaculture technology and mode has become an inevitable trend for the future development of aquaculture. Based on computer vision and deep learning technology, the feeding behavior of fish schools can be monitored in real time, and the feeding state can be analyzed to achieve scientific and precise feeding decisions.

[0003] Compared with the traditional artificial feeding method, intelligent feeding technology can effectively reduce feed waste, reduce breeding costs, improve breeding efficiency, and reduce water pollution caused by residual feed. However, the current quantification of fish feeding state relies on subjective evaluation criteria, and the use of artificial annotation for data samples often has the problem of insufficient scientificity, which cannot achieve accurate discrimination of the hunger state of fish schools; at the same time, existing researches are mostly focused on single feature extraction, lacking systematic exploration of multi-source key feature fusion, which limits the in-depth mining and accurate discrimination of the complexity of fish feeding behavior. In addition, the existing deep learning model has large calculation amount and high energy consumption, which makes it difficult to realize low-energy consumption deployment in real aquaculture scenarios.

[0004] Therefore, how to accurately quantify and evaluate the feeding state of fish schools combined with feature fusion methods, and realize stable and efficient recognition and feeding control on the low-energy consumption edge of high-density aquaculture farms, has become a key problem to be solved in the current field of intelligent aquaculture. SUMMARY

[0005] The present application aims to solve the problems of inaccurate discrimination of fish feeding state and high energy consumption of existing models in the prior art, and proposes an intelligent feeding system for cultured fish based on multi-feature fusion and a control method thereof, which can realize real-time, accurate and efficient recognition and decision control of the feeding state of fish schools, and realize intelligent feeding on a low-cost edge computing platform.

[0006] The technical scheme of the present application is: The present application provides an intelligent feeding system for cultured fish based on multi-feature fusion, comprising: an image acquisition module, configured to acquire a real-time video stream of the fish school; a space-motion feature extraction module, configured to obtain a depth optical flow image reflecting the feeding state feature of the fish school, and convert the depth optical flow image into an HSV color space, further extract a global average brightness meanV value of the image, determine the feeding state of the fish school, and classify the fish school feeding state image sample according to an adaptive threshold value to form a space-motion optical flow feature map data set; a motion trajectory feature extraction module, configured to extract the swimming trajectory of the fish, and form a swimming trajectory feature map data set; a feeding state recognition module, configured to extract features of the space-motion optical flow feature image data and the swimming trajectory feature image data, respectively, first for training to obtain optimal model weights, and second for real-time recognition through calculation after loading the optimal training weights, and the recognition result is transmitted to a feeding control module in the form of a classification instruction; a feeding control module, configured to receive the classification instruction and realize control of different feeding amounts; a feedback adjustment module, configured to evaluate the feeding state of the fish school after feeding is completed, and adjust the feeding amount or optimize the feeding decision.

[0007] Further, an integrated control board is further included, which encapsulates the space-motion feature extraction module, the motion trajectory feature extraction module, the feeding state recognition module, the feeding control module, the feedback adjustment module, and a data storage unit and an edge computing unit, and is externally connected with the image acquisition module through a standard data interface, and is externally connected with an automatic feeder through the feeding control module.

[0008] Further, the image acquisition module includes a camera device, a TCP communication interface and a standard data interface, the camera device is configured to acquire panoramic images of the water environment of the breeding pond, and the standard data interface and the TCP communication interface are connected with the camera device for data transmission and communication. During the feeding process, the image acquisition module records the real-time video stream of the fish school activity through the high-definition camera of the camera device, and transmits the real-time video stream to the image data storage unit.

[0009] Further, the space-motion feature extraction module includes a FlowNet2.0 depth optical flow model, a color space conversion unit, a feature value calculation unit and a data set division unit; the FlowNet2.0 is composed of a sub-network FlowNetC, a FlowNetSD and a plurality of FlowNetS stacked in sequence; The space-motion feature extraction module receives the video stream, and uses the FlowNet2.0 depth optical flow model to extract key features of the feeding state of the fish school in the image, and the steps are as follows: for any two consecutive video frames input and FlowNet2.0 extracts multi-scale feature maps from the input images and produces pixel-level feature descriptions With ; The correlation layer is introduced in the FlowNetC subnetwork, i.e. at each pixel position, the dot product similarity with all positions within a preset search window is calculated, as shown in equation (1): ; wherein x, y and u, v respectively represent the horizontal and vertical position coordinates of the pixels in the image; In FlowNet2.0, the FlowNetC subnetwork outputs a set of similarity heat maps through the correlation layer to capture the feature matching relationship between the image pairs. In the FlowNetS, FlowNet-SD and subsequent stacked subnetworks, the full convolution structure of the encoding-decoding type is adopted, and according to the input image or the warped image and the current optical flow estimation, the optical flow field is output through the multi-layer encoding-decoding full convolution structure. The optical flow field is usually represented as: ; wherein u(x, y) and v(x, y) respectively represent the horizontal and vertical motions at the pixel position (x, y); In the network model training, the loss function is supervised at multiple scales using the pixel end point error, as shown in equation (3): wherein S represents the index of the subnetwork or the multi-scale layer, α s represents the loss weight coefficient output by each layer or each subnetwork, represents the optical flow field predicted by the s-th network or scale, represents the true value of the optical flow field corresponding to the s-th scale; FlowNet2.0 concatenates the above subnetworks, each level uses the current flow field estimation to warp the second frame feature / image in space, and then learns the residual flow, forming a fine-tuning from coarse to fine, and finally outputs a high-precision optical flow map that takes into account both large and small displacements; The color space conversion unit converts the optical flow map from the BGR color space to the HSV color space, and the specific steps are as follows: First, normalize the three channel values in the RGB image to [0, 1]; second, calculate the maximum value and the minimum value​ and the difference value hue H The channel is expressed in angle, ranging from [0°, 360°], and the calculation is determined by the channel with the largest value, and the conversion formula is shown in equation (8): saturation S The channel is the ratio of the difference value to the maximum value, and the conversion formula is shown in equation (9): brightness V It is represented by the maximum value of the color channel, and the conversion formula is (10): The feature value calculation unit obtains the brightness by solving and extracting the three channels respectively, and comparing the features with the optical flow image. V The channel represents the spatial-motion fusion characteristics of the fish school; the global average brightness mean V of the channel is obtained. The time sequence curve of the fish feeding state is obtained by arranging the above feature values in time sequence, and the division threshold of the four fish feeding states of non-feeding, pre-food expectation, feeding and post-food expectation is determined according to the mean V value size and turning point position at each time in the curve, and the corresponding fish feeding state image sample is classified into four categories by the data set division unit to form a spatial-motion optical flow feature map data set, which is saved in the data storage unit for model training.

[0010] Further, the motion trajectory feature extraction module includes a YOLOv8-SHViT-Deepsort multi-target detection and tracking algorithm. The algorithm is divided into two stages: First, for the input video stream, the multi-target detection of the cultured fish school is realized through the YOLOv8-SHViT model. The attention part is calculated by the visual transformer general attention mechanism method, and the ; Non-attention part is transmitted through residual connection; in the feature splicing part, the SHSA module in the SHViT algorithm is introduced to fuse the part after attention mechanism and the non-attention part connected by residual connection, and multiply the projection weight for calculation, as shown in equation (12): Wherein, represents the feature after attention mechanism, ​​​​​a feature representing a residual connection; After the above calculation, the multi-target detection result of the cultured fish school is output by the YOLOv8-SHViT model; Secondly, the detection result output above is taken as the input of the Deepsort algorithm to realize multi-target tracking of the cultured fish school; For each tracked target, Deepsort uses a Hungarian filter to predict the position of the next frame in an 8-dimensional state space, and the state vector is usually The center coordinates (x, y), the aspect ratio a, the height h, and the respective speed are given the priori estimation of the current frame according to the state and covariance of the last frame; then, based on the Kalman prediction covariance, the Mahalanobis distance is used to measure the position matching degree of each predicted trajectory and the detection box, and the motion cost matrix is calculated; the cosine distance is used to measure the similarity of the trajectory history appearance vector and the current detection appearance vector, and the appearance cost matrix is calculated; and the two costs are fused according to the weight, and the Hungarian algorithm is used to solve the optimal assignment on the comprehensive cost matrix to pair the detection box with the existing trajectory one by one; Through the above algorithm, the swimming trajectory of all individuals in the fish school is extracted, and through time matching and alignment with the space-motion optical flow feature map dataset, a swimming trajectory feature map dataset is formed and saved in the data storage unit for model training.

[0011] Further, the feeding state recognition model adopts a Dual-EfficientViM-LSGA lightweight attention mechanism model, which is composed of a dual-branch lightweight model backbone EfficientViM, a lightweight Gaussian self-attention LSGAttention unit, and an MSF multi-stage hidden state fusion unit to form an image feature extraction network backbone. The dual-branch structure respectively extracts and fuses the fish school space-motion optical flow feature map dataset and the swimming trajectory feature map dataset, and the specific steps are as follows: The EfficientViM feature extraction layer obtains a feature map T1, which is first passed through a Patch Merging layer to obtain a feature map of H / 32 × W / 32 × D2 dimension, and then the obtained feature map is input into the EfficientViM module to promote information aggregation and channel interaction through a stacked HSM-SSD layer and a feedforward network, and a feature map T2 is obtained. The feature maps T1 and T2 are jointly input into the MSF multi-stage hidden state fusion module, and for each , a global representation is calculated by simply averaging the hidden states, as shown in equation (14): ; wherein, represents the i-th hidden state, N represents the total number of hidden states, represents the average hidden state value calculated; A 2-D Gaussian function is used to represent the spatial relationship of the image, and the Gaussian absolute position information is obtained to replace the relative position deviation, and the 2-D Gaussian function is defined as formula (15): ; wherein, is the standard deviation, represents the spatial position coordinates; the Gaussian absolute position replaces the relative position deviation, and the LSGA is represented as formula (16): ; wherein, Q represents a query matrix, X represents an input feature matrix, G represents a Gaussian position matrix, and d represents a feature dimension; The LSGAttention feature extraction layer first obtains a feature map of H / 64×W / 64×D3 dimensions through the Patch Merging layer, and then inputs the feature map into the LSGA unit and the MLP feedforward network for global and local information perception to obtain a feature map T3, complete feature extraction, and perform double-branch feature fusion. After fusing the double-branch image features, first, the optimal model weight is trained, and second, the model weight is loaded for real-time recognition in the edge computing unit. The recognition result is transmitted to the feeding control module in the form of a classification instruction to drive the servo motor to perform accurate feeding.

[0012] Further, the feeding control module comprises an instruction encoder, a downstream control circuit, and a servo motor driver, wherein the downstream control circuit comprises an instruction decoder, a PWM signal generator, a GPIO driving chip, and an interface. After receiving the classification instruction, the signal encoder encodes the instruction into an RS485 digital signal and transmits it to the downstream control circuit. The downstream control circuit analyzes the state code and generates a control instruction to the PWM signal generator, outputs a pulse waveform with adjustable duty cycle, and connects the servo motor driver through the GPIO driving chip and the interface. The control of different feeding amounts is realized by controlling the running time of the servo motor.

[0013] Further, the feedback adjustment module identifies and feeds back the feeding state after a single feeding action through the feeding state recognition model, adjusts the feeding amount or makes a new round of feeding decision.

[0014] The application further provides a multi-feature fusion-based intelligent feeding control method for farmed fish, which is applied to the multi-feature fusion-based intelligent feeding system for farmed fish. (1) Obtain the fish activity video stream, extract the space-motion optical flow feature map of the fish feeding state by using the FlowNet2.0 deep optical flow model, and track the fish individuals and extract the swimming trajectory feature map by using the YOLOv8-SHViT-Deepsort multi-target detection and tracking algorithm; (2) Extract the key feature values in the space-motion optical flow feature map by using the image space conversion and channel separation technology, and generate the time sequence curve for quantization, adaptively determine the fish feeding state threshold according to the feature value size and the turning point position, realize the classification of the image sample, and construct the fish space-motion optical flow feature map and the swimming trajectory feature map accurate quantization data set according to the alignment principle in time; (3) The above data set is input into two branches of the Dual-EfficientViM-LSGA feeding state recognition algorithm for training and obtaining the optimal model weight, and the recognition algorithm loaded with the optimal weight is used for edge computing unit to realize real-time recognition; (4) The recognition result is transmitted to the feeding control module in the form of classification instruction, and the servo motor is driven to execute accurate feeding; after single feeding is completed, the feedback adjustment module reevaluates the fish feeding state, and further optimizes the feeding decision.

[0015] The beneficial effects of the present application are as follows: The present application provides an intelligent feeding system for farmed fish based on multi-feature fusion and a control method thereof, which has excellent energy optimization capability on a low-cost edge computing platform, ensures the accuracy, efficiency and low-energy consumption operation of the fish feeding decision, and is suitable for various intensive farming environments. By using the system of the present application, scientific classification and recognition of the fish feeding state in a high-density recirculating aquaculture environment can be realized, so as to realize on-demand feeding according to the feedback of the feeding state. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The overall flowchart of the intelligent feeding system provided by the present application is shown in the figure; Figure 2 The space-motion feature extraction module flowchart of the intelligent feeding system provided by the present application is shown in the figure; Figure 3 The structure diagram of the FlowNet2.0 deep optical flow model of the intelligent feeding system provided by the present application is shown in the figure; Figure 4 The structure diagram of the YOLOv8-SHViT-Deepsort multi-target detection and tracking model of the intelligent feeding system provided by the present application is shown in the figure; Figure 5 The structure diagram of the SHViT module of the intelligent feeding system provided by the present application is shown in the figure; Figure 6A Deepsort multi-target tracking model structure diagram of the intelligent feeding system is provided in the present application. Figure 7 A feeding control module flow diagram of the intelligent feeding system is provided in the present application. Figure 8 A feedback adjustment module flow diagram of the intelligent feeding system is provided in the present application. Figure 9 A structure diagram of the Dual-EfficientViM-LSGA lightweight algorithm model is provided in the present application. Figure 10 A structure diagram of the EfficientViM attention mechanism unit is provided in the present application. Figure 11 A structure diagram of the LSGAttention lightweight attention mechanism unit is provided in the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0018] In order to further understand the present application, the present application will be further described in conjunction with the drawings and embodiments.

[0019] As shown in Figure 1 The present application provides an intelligent feeding system for farmed fish based on multi-feature fusion, which comprises an image acquisition module, a spatial-motion feature extraction module, a motion trajectory feature extraction module, a feeding state recognition module, a feeding control module, a feedback adjustment module and an integrated control board.

[0020] The spatial-motion feature extraction module, the motion trajectory feature extraction module, the feeding state recognition module, the feeding control module and the feedback adjustment module are packaged on the integrated control board, as well as a data storage unit and an edge computing unit. The image acquisition module is connected through a standard data interface, and the automatic baiting machine device is connected through the feeding control module.

[0021] The image acquisition module comprises a camera device (including a high-definition camera), a TCP communication interface and a standard data interface.

[0022] The camera device is arranged above the circulating water aquaculture pond, and the setting height and distance should be ensured to just cover the entire aquaculture pond to obtain panoramic image of the water environment of the aquaculture pond, and the standard data interface and the TCP communication interface are arranged on the integrated control version to ensure connection with the camera device for data transmission and communication.

[0023] In the actual feeding control process, the image acquisition module records the real-time video stream of fish activity through the high-definition camera, and transmits the video stream to the image data storage unit through the standard data interface.

[0024] As shown in Figure 2 , the space-motion feature extraction module includes a FlowNet2.0 deep optical flow model, a color space conversion unit, a feature value calculation unit, and a data set division unit.

[0025] The space-motion feature extraction module receives the video stream in the data storage unit, performs dimensionality reduction extraction, qualitative feature conversion, quantitative feature extraction, etc. on the fish feeding state features, and performs scientific data set division. The following steps are included: Step 1, feature dimensionality reduction extraction.

[0026] The FlowNet2.0 deep optical flow model is used to extract the key features of the fish feeding state (fish activity level and water surface texture) in the image, and generate a model to calculate the space-motion feature optical flow map of the fish feeding state.

[0027] As shown in Figure 3 , FlowNet2.0 is composed of subnetworks FlowNetC, FlowNetSD, and a number of FlowNetS stacked in sequence, relying on convolutional neural networks to automatically learn the mapping from pixels to optical flow. The correlation layer realizes explicit feature matching, and "end-to-end regression" replaces the traditional optical flow pyramid and optimization. Stacking + warping completes the coarse-to-fine joint optimization, thereby efficiently and accurately extracting the optical flow image.

[0028] First, for any two consecutive video frames and , FlowNet2.0 extracts multi-scale feature maps through a series of convolutional layers, producing pixel-level feature descriptions and ; in the FlowNetC subnetwork, a correlation layer is introduced: at each pixel position of , the dot product similarity with at all positions within a preset search window (e.g. DxD) is calculated, as shown in equation (1): where x, y represent the horizontal and vertical position coordinates of the pixels in the image, respectively;u, v represent the displacement in the horizontal and vertical directions respectively.

[0029] The correlation layer outputs a set of similarity heat maps, which clearly indicate the most likely displacement direction and amplitude of each pixel between frames, providing a "rough matching" basis for the subsequent convolutional network.

[0030] In FlowNetS, FlowNet-SD, and subsequent stacked subnetworks, the network directly accepts the original image (or distorted image) splicing / feature + correlation map, and after a multi-layer encoding-decoding full convolutional structure, regresses the output optical flow field, as shown in formula (2): in, u(x, y) and v(x, y) Represents the horizontal and vertical motion at the pixel position (x,y), respectively.

[0031] In network model training, the loss function loss function Pixel endpoint error is used for supervision at multiple scales, as shown in formula (3): in, S represents the index of a subnetwork or multi-scale layer, α s Represents the loss weight coefficient of each layer or each sub-network output, represents the optical flow field predicted by the sth network or scale, Represents the true value of the optical flow field corresponding to the s-th scale; FlowNet2.0 connects the above sub-networks in series. At each level, the features / image of the second frame are spatially distorted based on the current flow field estimate, and then the residual optical flow is learned to form a coarse-to-fine fine-tuning, and finally output a high-precision optical flow map that takes into account both large and small displacements.

[0032] Through the above steps, a deep optical flow image is obtained that reflects the feeding state of the fish school. In this optical flow image, the location of the optical flow represents the spatial position information, the different optical flow colors represent the movement direction information, and the brightness of the optical flow represents the movement amplitude of the fish school.

[0033] Step 2: Qualitative feature conversion.

[0034] The color space conversion unit converts the extracted deep optical flow image from the BGR color space to the HSV color space, where H represents the image hue, S represents the image saturation, and V represents the image brightness. The specific steps of the conversion are: First, the three channel values ​​in the RGB image range from [0, 255]. First, they must be normalized to [0, 1], then we have formula (4): Secondly, the maximum value of the normalized color channel Minimum and the difference Calculations are performed, and we have formula (5), formula (6) and formula (7): tone H The channel represents the type of color and is expressed in angles, ranging from [0°, 360°]. Its calculation is determined by the channel with the largest value. The conversion formula is shown in formula (8): Saturation S The channel represents the purity or vividness of the color and is defined as the ratio of the difference to the maximum value. The conversion formula is shown in formula (9): brightness V Indicates the brightness of the color, which is represented by the maximum value of the color channel. Its conversion formula is (10): Step 3: Quantitative characterization and extraction.

[0035] The eigenvalue calculation unit solves and extracts the three channels respectively and compares the features with the optical flow image respectively. It is found that the motion-related features are mainly reflected in the brightness of the optical flow image. V In the channel, this channel reflects the movement speed and spatial position of the fish school, representing the spatial-motion fusion characteristics of the fish school. The characteristic values ​​of this channel are globally averaged to obtain the global average brightness mean at each moment. V value.

[0036] Step 4: The time series curve of the fish feeding state is obtained by the above method, and the mean value of each moment in the curve is used to calculate the feeding state of the fish. V The value and turning point position are adaptively determined to determine the four feeding states of fish: non-feeding, pre-food anticipation, feeding, and post-food anticipation. Figure 2 , obtain the adaptive feeding state division threshold, the data set division unit classifies the image into four classification corresponding data set storage units, and forms a space-motion feature optical flow map data set for model training.

[0037] The dataset storage unit is divided into the following four folders according to the feeding status of the fish: non-feeding state: 0 (dataset folder_NF); pre-food anticipation state: 1 (dataset folder_PFE); feeding state: 2 (dataset folder_FD); post-food anticipation state: 3 (dataset folder_AFE).

[0038] Among them, the improved four fish feeding states are objectively described as follows: in the non-feeding state, the fish has no obvious feeding desire and is distributed in a scattered or ring-shaped manner in the pool; in the food expectation state, the fish has obvious feeding desire and is gathered near the feeding point, wherein the former expectation state is the food expectation generated by the hunger of the fish, and the latter expectation is the food expectation generated by the unsatisfied fish; in the feeding state, the fish is actively feeding, and the water surface is splashing.

[0039] The motion trajectory feature extraction module includes a YOLOv8-SHViT-Deepsort multi-target detection and tracking algorithm, and the proposed YOLOv8-SHViT-Deepsort algorithm is used for multi-target detection and tracking of the cultured fish population. The algorithm is divided into two stages: first, for the input video stream, the multi-target detection of the cultured fish population is realized through the YOLOv8-SHViT model; second, the output detection result is taken as the input of the Deepsort algorithm to realize the multi-target tracking of the cultured fish population, so as to generate the swimming trajectory of the fish population individual, and then the swimming trajectory feature map of the fish population individual is extracted, and time matching is performed with the spatial-motion optical flow feature map dataset to form the swimming trajectory feature map dataset.

[0040] As shown in Figure 4 , the present application introduces a YOLOv8 lightweight and efficient detection model as the feature extraction backbone, which has improved accuracy and inference speed compared to previous versions. The structure mainly includes a backbone network, a neck network and a detection head. The backbone network is responsible for feature extraction, the neck network is responsible for feature fusion to improve detection capability, and the detection head is responsible for final target classification and regression.

[0041] In particular, the YOLOv8 baseline model is improved by adding a SHViT module in the model backbone network. This module further enhances the feature extraction capability through the fusion of global and local information, while the lightweight and efficient architecture design can reduce excessive information redundancy, especially suitable for fast operation on edge devices.

[0042] As shown in Figure 5 , the core layer in the SHViT module is a single-head attention (SHSA) layer, which only performs attention calculation on part of the channels, but the final projection operation is applied to all channels to ensure that the attention features can be effectively propagated to the remaining channels. The attention part (i.e. the rC part in the figure) is still calculated by the Vision Transformer (Vision Transformer) general attention mechanism method, as shown in equation (11): Where Q represents the query matrix, K represents the key matrix, and V represents the value matrix.

[0043] The non-attention part (i.e., the (1-r)C part in the figure) is transmitted through a residual connection.

[0044] In the feature splicing part, the part after the attention mechanism is fused with the non-attention part connected by the residual connection, and multiplied by the projection weight , as shown in equation (12): wherein, represents the feature after the attention mechanism, represents the feature connected by the residual connection; represents the projection weight.

[0045] After the above calculation is completed, the output projection operation is performed, and the feature extraction process of this stage is completed. Finally, the aquaculture fish group multi-target detection result is output by the YOLOv8-SHViT model, and the result is used as the input of the Deepsort multi-target tracking algorithm.

[0046] As shown in Figure 6 , the Deepsort multi-target tracking algorithm uses a set of bounding boxes and their class confidence output by the YOLOv8-SHViT model after efficient detection to locate the target to be tracked. For each detected target, its image region is cropped and sent to the pre-trained ReID network to extract a fixed-dimensional appearance description vector, which is used to distinguish different targets with similar appearances in subsequent association. For each tracked target, Deepsort uses the Hungarian filter (KalmanFilter) to predict the position of the next frame in an 8-dimensional state space, and the state vector is usually : center coordinates x, y , aspect ratio a , height h , and their respective speeds. According to the state and covariance of the previous frame, the prior estimate of the current frame is given. Subsequently, based on the Kalman prediction covariance, the Mahalanobis distance is used to measure the position matching degree between each predicted trajectory and the detection box, and the motion cost matrix is calculated; the cosine distance is used to measure the similarity between the historical appearance vector of the trajectory and the current detection appearance vector, and the appearance cost matrix is calculated. And the two cost matrices are fused according to the weight, as shown in equation (13): wherein, represents the Mahalanobis distance, represents the cosine distance, is the weight coefficient.

[0047] The Hungarian algorithm (Hungarian) is used to solve the optimal assignment on the comprehensive cost matrix to pair each detection box with an existing trajectory.

[0048] For detections that do not match any track, a new track is initiated and set to a "tentative" state, which is converted to a formal track only after a few consecutive frames (e.g. 3 frames) of consistent matches; for tracks that are not matched for a number of consecutive frames (e.g. 30 frames), they are considered "lost" and deleted, releasing the ID.

[0049] For each pair of matched track and detection, the Kalman filter state is updated using the observation of the detection box, and the newly extracted appearance vector is incorporated into the appearance model of the track in a momentum update or feature gallery manner; for unmatched tracks, only Kalman prediction is performed without observation update, keeping the uncertainty rising.

[0050] Through the above algorithm, the swimming tracks of all individuals in the cultured fish school are extracted, and through the data set division unit and the space-motion optical flow feature data set, the time matching alignment is performed to form the swimming track feature map data set of the feeding state of the cultured fish school.

[0051] As shown in Figure 9 , the feeding state recognition model takes the classified data set file as input, and pre-trains the data set using the designed Dual-EfficientViM-LSGA algorithm. The designed deep learning image recognition algorithm combines the lightweight feature extraction unit, the main trunk of EfficientViM, the lightweight self-Gaussian attention LSGAttention unit, and the MSF multi-stage hidden state fusion unit to form the image feature extraction network trunk.

[0052] In particular, the feature extraction trunk of the model adopts a double-branch structure design, which respectively extracts features from the space-motion optical flow feature data set and the swimming track feature map data set, and fuses the two features at the end.

[0053] As shown in Figure 10 , the lightweight feature extraction unit of EfficientViM performs channel mixing based on its HSM-SSD layer by introducing a shared global hidden state h, which includes gating and output projection operations, directly on the reduced dimension latent array to reduce computational complexity.

[0054] First, the input hidden state h in is calculated, and then the linear projection is used to convert h in into the global hidden state h, thereby reducing the computational complexity from O(LD 2 ) to O(ND 2 ), where L is the sequence length, D is the channel number, and N is the state number (and N << D). By reducing the number of hidden states to handle the computationally intensive projection task, the overall computational cost mainly depends on the state number rather than the channel number.

[0055] As Figure 9 shown, the designed image recognition algorithm is mainly divided into four stages in the feature extraction process: Stage I: This stage is the EfficientViM feature extraction layer. First, through 4 consecutive 3x3 convolution layers, the input image of Hx Wx3 is mapped to a feature map of H / 16x W / 16x D1 dimension, and then the obtained feature map is input into the EfficientViM module to promote information aggregation and channel interaction through the stacked HSM-SSD layer and the feedforward network, and the feature map T1 is obtained. Stage II: This stage is also the EfficientViM feature extraction layer. The feature map series T1 is first obtained through the PatchMerging layer to obtain a feature map of H / 32x W / 32x D2 dimension, and then the obtained feature map is input into the EfficientViM module to promote information aggregation and channel interaction through the stacked HSM-SSD layer and the feedforward network, and the feature map T2 is obtained. Stage III: The feature maps T1 and T2 are jointly input into the MSF multi-stage hidden state fusion module, which fuses the hidden states from multiple stages of the network to generate a prediction logic probability, thereby integrating low-level and high-level features and enhancing the generalization ability during reasoning. For each , a global representation is calculated by simply averaging the hidden states, as shown in equation (14): wherein, represents the i-th hidden state, N represents the total number of hidden states, represents the average hidden state value calculated.

[0056] Referring to Figure 11 , the lightweight self-Gaussian attention unit (LSGAttention) combines the Gaussian position bias and feature dimension reduction mechanism, realizes the lightweight attention mechanism through adaptive weighting and efficient matrix operation, and uses a 2-D Gaussian function to represent the spatial relationship of the image to obtain the Gaussian absolute position information. The relative position deviation is replaced by the Gaussian absolute position, and the 2-D Gaussian function is defined as equation (15): wherein, is the standard deviation, represents the spatial position coordinates, and t represents the number of pixel points extracted in each image block.

[0057] The Gaussian absolute position replaces the relative position deviation, and the LSGA can be represented as equation (16): Wherein, Q represents a query matrix, X represents an input feature matrix, G represents a Gaussian position matrix, and d represents a feature dimension.

[0058] Stage IV: This stage is the LSGAttention feature extraction layer. The fused features are first passed through the PatchMerging layer to obtain a feature map with a dimension of H / 64 x W / 64 x D3, and then the feature map is input into the LSGA unit to perceive global and local information through the light self-gaussian-attention and MLP feedforward network, to obtain a feature map T3, complete feature extraction, and perform double-branch feature fusion.

[0059] After the fusion of the double-branch image features, the trainable feeding state recognition model outputs the image prediction result through the fully connected layer, and continues to update the weight parameters through the back propagation process of the model. Through the training-forward propagation-parameter updating-back propagation mode, the weight parameters are gradually fitted to the convergence state. At this time, the optimal weight parameter file for the recirculating water high-density aquaculture scene is obtained, and is used for the feeding state recognition model; As shown in Figure 1 , the feeding state recognition model carries the optimal weight file, and the fish school feeding state image input by the data storage module runs the edge computing unit to predict the actual fish school feeding state, and outputs the classification instruction of the fish school feeding state to the feeding control module; Different feeding amounts correspond to fish school feeding states: non-feeding state-no feeding process, food expected state (before and after)-20% of the single normal feeding amount, feeding state-single normal feeding amount. That is, the control logic of the feeding strategy of the present application is: no feeding when the fish school is not hungry, a small amount of exploratory feeding for the food expected state (hunger or unsatisfied) of the fish school, and normal feeding for the normal feeding state of the fish school.

[0060] The feeding control module includes an instruction encoder, a downstream control circuit, and a servo motor driver. The downstream control circuit includes an instruction decoder, a PWM signal generator, a GPIO driving chip, and an interface.

[0061] As shown in Figure 7 , the feeding control module receives the classification instruction of the deep learning recognition module, the signal encoder encodes the instruction into an RS485 digital signal, and transmits it to the downstream control circuit. The downstream control circuit analyzes the state code and generates a control instruction to the PWM signal generator, outputs a pulse waveform with adjustable duty cycle, and connects the servo motor driver through the GPIO driving chip and the interface. The 5V TTL level output of the 20-23 pin of the GPIO driving chip is output to the servo motor driver, and the control of different feeding amounts of the baiting machine is realized by controlling the running time of the servo motor.

[0062] As shown in Figure 8 The feedback adjustment module works as follows: after each feeding action is completed, the feeding state recognition model is reactivated, the feeding state after feeding is recognized and fed back through the feeding state recognition model, and feeding amount feedback adjustment or a new round of feeding decision judgment is started.

[0063] The present application can realize scientific classification and recognition of fish feeding state in high-density recirculating aquaculture environment, so as to realize on-demand feeding according to the feedback of the feeding state.

[0064] The present application also provides a control method applied to the above-mentioned intelligent feeding system for farmed fish based on multi-feature fusion, comprising the following steps: (1) Obtain the fish activity video stream, extract the space-motion optical flow feature map of the fish feeding state using the FlowNet2.0 deep optical flow model, and track the fish individuals and extract the swimming trajectory feature map through the YOLOv8-SHViT-Deepsort multi-target detection and tracking algorithm; (2) Extract the key quantitative feature values in the space-motion optical flow feature map through image space conversion and channel separation technology and generate a time series curve for quantification, adaptively determine the fish feeding state threshold according to the feature value size and turning point position, realize scientific classification of image samples, and construct the space-motion optical flow feature map and swimming trajectory feature map precise quantification dataset according to the time alignment principle; (3) Input the above two types of datasets into the two branches of the Dual-EfficientViM-LSGA feeding state recognition algorithm for training and obtain the optimal model weight, and the recognition algorithm loaded with the optimal weight will be used for edge computing unit to realize real-time recognition; (4) The recognition result is transmitted to the feeding control module in the form of a classification instruction to drive the servo motor to perform precise feeding; after a single feeding is completed, the feedback adjustment module re-evaluates the fish feeding state to further optimize the feeding decision.

[0065] The above description is only a preferred embodiment of the present application, and is not a limitation on the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements to some technical features. Any modification, equivalent replacement, modification, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An intelligent feeding system for farmed fish based on multi-feature fusion, characterized in that: include: Image acquisition module, used to obtain real-time video stream of fish schools; The spatial-motion feature extraction module is used to obtain a depth optical flow image reflecting the feeding characteristics of the fish school, convert the depth optical flow image into the HSV color space, and further extract the global average brightness meanV value of the image; determine the feeding state of the fish school and classify the fish feeding state image samples based on the adaptive threshold discrimination to form a spatial-motion optical flow feature map dataset; The motion trajectory feature extraction module is used to extract the swimming trajectory of fish and form a swimming trajectory feature map dataset; The feeding state recognition module is used to extract features from the spatial-motion optical flow feature image data and the swimming trajectory feature image data. First, it is used for training to obtain the optimal model weight. Second, after loading the optimal training weight, real-time recognition is achieved through calculation. The recognition results are transmitted to the feeding control module as classification instructions. Feeding control module, used to receive classification instructions and realize the control of different feeding amounts; The feedback adjustment module is used to evaluate the feeding status of fish after feeding is completed, adjust the feeding amount or optimize feeding decisions.

2. The intelligent feeding system for farmed fish based on multi-feature fusion according to claim 1, characterized in that: It also includes an integrated control board, which encapsulates a space-motion feature extraction module, a motion trajectory feature extraction module, a feeding state recognition module, a feeding control module, a feedback adjustment module, as well as a data storage unit and an edge computing unit. It is connected to an external image acquisition module through a standard data interface and to an external automatic bait feeder through a feeding control module.

3. The intelligent feeding system for farmed fish based on multi-feature fusion according to claim 1, characterized in that: The image acquisition module includes a camera device, a TCP communication interface and a standard data interface. The camera device is used to obtain a panoramic image of the aquaculture pond water environment. The standard data interface and the TCP communication interface are connected to the camera device for data transmission and communication. During the feeding process, the image acquisition module records the real-time video stream of the fish school activities through the high-definition camera of the camera equipment and transmits it to the image data storage unit.

4. The intelligent feeding system for farmed fish based on multi-feature fusion according to claim 1, characterized in that: The spatial-motion feature extraction module includes a FlowNet2.0 deep optical flow model, a color space conversion unit, an eigenvalue calculation unit, and a data set division unit; FlowNet2.0 is composed of sub-networks FlowNetC, FlowNetSD, and several FlowNetS stacked in sequence; The spatial-motion feature extraction module receives the video stream and uses the FlowNet2.0 deep optical flow model to extract the key features of the fish feeding state in the image. The steps are as follows: for any two consecutive video frames input and FlowNet2.0 extracts multi-scale feature maps from it and generates pixel-level feature descriptions and ; Introducing the correlation layer in the FlowNetC subnetwork, that is, At each pixel position, calculate The dot product similarity of all positions within a preset search window is shown in formula (1): ； in, x,y Respectively represent the horizontal and vertical position coordinates of the pixels in the image; u,v Represent the displacement in the horizontal and vertical directions respectively; In FlowNet2.0, the FlowNetC subnetwork outputs a set of similarity heat maps through the correlation layer to capture the feature matching relationship between image pairs. In FlowNetS, FlowNet-SD and subsequent stacked subnetworks, an encoding-decoding full convolutional structure is adopted. According to the input image or distorted image and the current optical flow estimation, after a multi-layer encoding-decoding full convolutional structure, the regression output optical flow field is usually expressed as: ； in u(x, y) and v(x,y) Represents the horizontal and vertical motion at the pixel position (x, y), respectively; In network model training, the loss function Pixel endpoint error is used for supervision at multiple scales, as shown in formula (3): in, S represents the index of a subnetwork or multi-scale layer, α s Represents the loss weight coefficient of each layer or each sub-network output, represents the optical flow field predicted by the sth network or scale, Represents the true value of the optical flow field corresponding to the s-th scale; FlowNet2.0 connects the above sub-networks in series. At each level, the current flow field estimate is used to spatially warp the features / image of the second frame. The residual optical flow is then learned to form a coarse-to-fine fine-tuning process. The final output is a high-precision optical flow map that takes into account both large and small displacements. The color space conversion unit converts the optical flow map from the BGR color space to the HSV color space. The specific steps are as follows: First, the three channel values ​​in the RGB image are normalized to [0, 1]. Second, the maximum and minimum values ​​of the normalized color channels and their differences are calculated to obtain the maximum value. Minimum and the difference ; tone H The channel is expressed in degrees, ranging from [0°, 360°]. Its calculation is determined by the channel with the largest value. The conversion formula is shown in formula (8): ； Saturation S The channel is the ratio of the difference to the maximum value, and the conversion formula is shown in formula (9): ； brightness V It is represented by the maximum value of the color channel, and its conversion formula is (10): ； The eigenvalue calculation unit solves and extracts the three channels respectively and compares the features with the optical flow image to obtain the brightness. V The channel represents the spatial-motion fusion characteristics of the fish school; the global average brightness mean of the channel is obtained V Eigenvalue; The above-mentioned characteristic values ​​are arranged in time series to obtain a time series curve of the fish feeding state. According to the size of the meanV value and the turning point position at each moment in the curve, the division thresholds of the four fish feeding states of non-feeding, pre-food anticipation, feeding, and post-food anticipation are determined. The data set division unit divides the corresponding fish feeding state image samples into four categories to form a space-motion optical flow feature map data set, which is saved in the data storage unit for model training.

5. The intelligent feeding system for farmed fish based on multi-feature fusion according to claim 1, characterized in that: The motion trajectory feature extraction module includes the YOLOv8-SHViT-Deepsort multi-target detection and tracking algorithm; The algorithm is divided into two stages: First, for the input video stream, the YOLOv8-SHViT model is used to detect multi-target farmed fish. The attention part is calculated by the general attention mechanism method of the visual transformer to obtain ; The non-attention part is transmitted through residual connection; In the feature splicing part, the SHViT algorithm is introduced SHSA Module, which fuses the part after the attention mechanism with the non-attention part of the residual connection and combines it with the projection weight Multiplication calculation, as shown in formula (12): ； in, Represents the features after the attention mechanism, Represents the characteristics of the residual connection; After the above calculations, the YOLOv8-SHViT model is used to output the multi-target detection results of farmed fish. Secondly, the above output detection results are used as the input of Deepsort algorithm to achieve multi-target tracking of farmed fish; For each tracked target, Deepsort uses the Hungarian filter to predict the next frame position in the 8-dimensional state space, and its state vector is usually The center coordinates (x, y), aspect ratio a, height h, and respective velocities are estimated a priori for the current frame based on the state and covariance of the previous frame. Subsequently, the Mahalanobis distance is used to measure the positional match between each predicted trajectory and the detection box based on the Kalman prediction covariance, and a motion cost matrix is ​​calculated. The cosine distance is used to measure the similarity between the trajectory's historical appearance vector and the current detection appearance vector, and an appearance cost matrix is ​​calculated. The two costs are then weighted and fused, and the Hungarian algorithm is used to solve the optimal assignment on the combined cost matrix, pairing the detection boxes with the existing trajectories one-to-one. The swimming trajectories of all individuals in the fish school are extracted through the above algorithm, and the swimming trajectory feature map dataset is formed by temporal matching and alignment with the spatial-motion optical flow feature map dataset, which is saved in the data storage unit for model training.

6. The intelligent feeding system for farmed fish based on multi-feature fusion according to claim 1, characterized in that: The feeding state recognition model adopts the Dual-EfficientViM-LSGA lightweight attention mechanism model, which consists of a dual-branch lightweight model backbone EfficientViM, a lightweight Gaussian self-attention LSGAttention unit, and an MSF multi-stage hidden state fusion unit to form an image feature extraction network backbone; The dual-branch structure extracts and fuses features from the fish school space-motion optical flow feature map dataset and the swimming trajectory feature map dataset respectively. The specific steps are as follows: The EfficientViM feature extraction layer obtains feature map T1, which is first passed through the Patch Merging layer to obtain a feature map of dimension H / 32 × W / 32 × D2. The obtained feature map is then input into the EfficientViM module. The stacked HSM-SSD layer and feedforward network promote information aggregation and channel interaction to obtain feature map T2. The feature maps T1 and T2 are input into the MSF multi-stage hidden state fusion module. , calculate a global representation by The simple average of the hidden state is as follows: ； in, represents the i-th hidden state, N represents the total number of hidden states, represents the calculated average hidden state value; The 2-D Gaussian function is used to represent the spatial relationship of the image, and the Gaussian absolute position information is obtained. The Gaussian absolute position is used to replace the relative position deviation. The 2-D Gaussian function is defined as follows: ;in, is the standard deviation, Represents the spatial position coordinates; the Gaussian absolute position replaces the relative position deviation, and LSGA is expressed as follows: ; Where Q represents the query matrix, X represents the input feature matrix, G represents the Gaussian position matrix, and d represents the feature dimension; The LSGAttention feature extraction layer first passes the fused features through the Patch Merging layer to obtain a feature map of H / 64 × W / 64 × D3 dimensions. This feature map is then input into the LSGA unit and MLP feedforward network for global and local information perception, resulting in feature map T3. This completes feature extraction and performs dual-branch feature fusion. After fusing the dual-branch image features, the optimal model weights are first trained, and then the model weights are loaded into the edge computing unit for real-time recognition. The recognition results are transmitted to the feeding control module in the form of classification instructions, driving the servo motor to perform precise feeding.

7. The intelligent feeding system for farmed fish based on multi-feature fusion according to claim 1, characterized in that: The feeding control module includes a command encoder, a downstream control circuit, and a servo motor driver, wherein the downstream control circuit includes a command decoder, a PWM signal generator, a GPIO driver chip and an interface; After the feeding control module receives the classification instruction, the signal encoder encodes the instruction into an RS485 digital signal and passes it to the downstream control circuit. The downstream control circuit parses the state code and generates a control instruction, which is passed to the PWM signal generator, outputs a pulse waveform with an adjustable duty cycle, and connects to the servo motor driver through the GPIO driver chip and interface. The control of different feeding amounts is achieved by controlling the servo motor running time.

8. The intelligent feeding system for farmed fish based on multi-feature fusion according to claim 1, characterized in that: After a single feeding action is completed, the feedback adjustment module identifies and provides feedback on the feeding state after feeding through the feeding state recognition model, and adjusts the feeding amount or makes a new round of feeding decisions.

9. A method for intelligent feeding control of farmed fish based on multi-feature fusion, characterized in that: The intelligent feeding system for farmed fish based on multi-feature fusion as described in any one of claims 1 to 8 comprises the following steps: (1) Obtain a video stream of fish activity, use the FlowNet2.0 deep optical flow model to extract the spatial-motion optical flow feature map of the fish feeding state, and use the YOLOv8-SHViT-Deepsort multi-target detection and tracking algorithm to track individual fish and extract the swimming trajectory feature map; (2) Through image space conversion and channel separation technology, key eigenvalues ​​in the spatial-motion optical flow feature map are extracted and time series curves are generated for quantification. The threshold of the fish feeding state is adaptively determined according to the size of the eigenvalue and the position of the turning point to achieve the classification of image samples. Based on the principle of time alignment, accurate quantification datasets of the fish spatial-motion optical flow feature map and the swimming trajectory feature map are constructed respectively; (3) The above datasets are respectively input into the two branches of the Dual-EfficientViM-LSGA feeding state recognition algorithm for training and obtaining the optimal model weights. The recognition algorithm equipped with the optimal weights will be used in the edge computing unit to achieve real-time recognition; (4) The recognition results are transmitted to the feeding control module in the form of classification instructions, which drives the servo motor to perform precise feeding. After a single feeding is completed, the feedback adjustment module re-evaluates the feeding status of the fish school and further optimizes the feeding decision.

Images